The tuberculosis task [1] of ImageCLEF 2017 Challenge [2] considers two subtasks both dealing with 3D CT images. The subtask #1 is dedicated to the problem of single image-based distinguishing between multi-drug resistant tuberculosis (MDR TB) cases and drug sensitive (DS) ones. The task itself is very challenging and so far there are no techniques reported which allow robust and accurate prediction of tuberculosis drug resistance based solely on lung CT images. Several studies reported the statistically significant links between presence of visually detected radiological findings and drug resistance status [3,4]. Also some research was carried out to detect statistically significant links between the drug resistance and structural features of radiological images of lung [5] and some trials were made to assess the possible prediction accuracy [6]. However in those studies the datasets used were not large enough and contained relapse tuberculosis cases which have much higher probability of begin drug-resistant. Instead, the dataset collected for ImageCLEF 2017 tuberculosis subtask #1 contained only DS and MDR cases without transitional single-drug resistant and poly-drug resistant ones and no relapses in order to make the dataset as unbiased as possible.
For extraction of lung regions in both subtasks, a domestic implementation of a conventional approach of segmentation using non-rigid image registration was used instead of the one proposed by the organizers. In our case the method utilized 130 reference CT scans with manually segmented lungs. These 130 CT scans represent completely separate dataset and have no respect to any of Im-ageCLEF tuberculosis datasets. For each target CT scan, a similarity measure was calculated between the target image and the reference images and top-5 most similar reference images were selected. The selected images along with the corresponding lung masks were registered using 'elastix' software tool [7], the final segmentation mask was obtained by means of averaging. The implemented method demonstrated high robustness to the presence of large lesion in lungs (see Fig. 1).
For subtask #1 we used an image description method which operates with socalled image supervoxels which are basically a 3D-version of conventional 2D image superpixels [8]. The method considers categorization of image supervoxels into classes according to a precalculated supervoxel dictionary similarly to the well known bag-of-words approach [9]. For composing a meaningful supervoxels dictionary, an auxiliary image dataset was composed. The dataset included 229 small 3D CT image regions of size 128×128×128 voxels extracted from CT scans. Before extraction of regions, the original CT images were re-sampled using nearest-neighbor interpolation in order to equalize sizes of voxel along all the three axes, i.e. make them cubic-shaped (see examples in Fig. 2).
For subtask #1, a method for quantitative description of biomedical images based on supervoxel representation and utilizing the co-occurrence concept was used. To our best knowledge the potential of superpixel/supervoxel-based image description has not been extensively researched yet [10].
The proposed image description method includes two major stages: (a) generating a supervoxel dictionary and (b) describing the images using the obtained dictionary. With this study, superpixel dictionaries were represented by the sets of features of the most typical supervoxels occurring on the images of a given type. The generating superpixel dictionary stage included the following steps:
selection of a certain number of representative images of given type; extraction of supervoxels from the selected images; extraction of supervoxels features; splitting the supervoxels feature-space into N clusters; calculating cluster (class) centroids; composing the supervoxel dictionary (set of centroids).
The image description stage was based on calculation of histograms and cooccurrence matrices of image supervoxels categorized into N classes according to the previously obtained dictionary. This included the following:
extraction of supervoxels from the target image; extraction of supervoxels features; categorization of each supervoxel into one of N classes according to the precalculated dictionary; calculating a co-occurrence matrix [11] of the categorized supervoxels.
In [12] the authors used superpixel dictionaries for semantic segmentation of street images. A set of 1708 superpixel features including color, texture, shape and location features was used for the task of scene description. In our study, we used a set of 6 major supervoxel features which basically describe texture and shape of a single supervoxel:
mean intensity of internal pixels; standard deviation of intensity; entropy of intensity; mean gradient magnitude; sphericity; -"cubeness". Sphericity was calculated using formula:
where V is the total number of voxels in supervoxel, and A is the number of border voxels in supervoxel. Sphericity value is close to 1 if the supervoxels shape is similar to sphere. "Cubeness" feature expressed the extent of how much the supervoxel shape is similar to cube and was calculated as:
where V bb corresponds to number of voxels in bounding box of the supervoxel considered. Maximum value of "cubeness" feature is 1 in case of ideally cubicshaped supervoxel. The meaning of this feature is dictated by the algorithm of generation of supervoxels. At the initialization step, the shape of all supervoxels is cubic. And if the underlying image substrate is enough homogeneous the resultant supervoxels remain cubic-shaped. A 3D version of the superpixel generation algorithm [8] used with this study has two control parameters: superpixel size Sz and a regularization parameter Reg. The examples of generated supervoxels are shown in Fig. 3. Supervoxel dictionaries were generated for several combinations of parameters Sz and Reg. Supervoxel clustering was performed using k-means algorithm, number of clusters being set to N = 8, 16, 32 and 64.
The proposed method was finally applied to the training set images of Image-CLEF tuberculosis subtask #1. The following combinations of control parameters of supervoxel extraction algorithm were used: Sz = 32, Reg = 0.1; Sz = 32, Reg = 0.3; Sz = 32, Reg = 1; Sz = 48, Reg = 0.1; Sz = 48, Reg = 0.3; Sz = 48, Reg = 1. The supervoxel dictionary size N was set to 8, 16, 32 and 64. Supervoxel maps were calculated for all of the 230 training CT scans, class numbers were assigned to supervoxels and for each image a supervoxels class co-occurrence matrix was calculated which was further used as a feature vector for prediction.
Only feature vector elements which were correlated with drug resistance at significance level p < 0.05 were selected for further prediction. Finally, assessment of patients drug resistance probability was performed with use of Logistic Regression classifier within 5-fold cross-validation procedure. Area under ROCcurve (AUC) was used as a measure of prediction performance. The results are shown in Table 3.3. According to the table of results, the best prediction performance was achieved for the following combination of parameters: Sz = 32, Reg = 1 and size of supervoxel dictionary N = 64. However, it should be notices that even the highest achieved performance is pretty low in general and with 230 study cases corresponds to statistical significance level of roughly p ∼ 0.01.
For this subtask, an extended multi-sort, multi-dimensional co-occurrence matrix approach was used which is proven to be powerful and flexible enough to capture a broad range of structural properties of both 2D and 3D medical images [13]. For describing lung CT image structure we employed the most general, six-dimensional matrices of IIGGAD type counting voxel pairs with certain intensities (I), gradient magnitudes (G), and mutual angles (A) between the gradient vectors. CT image intensity range from -1000 to +500 Hounsfield Units (HU) was quantized to 12 bins. Gradient values and angles between gradients were both quantized to 6 bins. The matrices consider co-occurrence of voxels in 3D on distances from 1 to 5 measured in axial voxel size. The values of algorithm parameters were selected empirically to maximize classification performance on training set of images. The resultant multidimensional co-occurrence matrices contained in total 12 2 × 6 2 × 6 × 5 = 155 520 elements, most of which were however zeros.
To reduce the dimensionality of the feature space, a Principal Component Analysis method (PCA) was applied and the first 50 PC's were considered for further analysis. The prediction of patient's tuberculosis type was performed with use of Random Forests classifier.
The evaluation of the proposed image classification method within 5-fold cross-validation procedure demonstrated classification accuracy of 57.0% and the un-weighted Cohens Kappa coefficient value was 0.442.
Since subtask #1 represent a very challenging and important problem, most of the efforts of our team was focused on drug resistance prediction subtask. Several different approaches were tested and various descriptor types were examined. Algorithms of automated detection of lesions [14] were used to derive additional information on the affected level of lungs. Some trials were made to utilize Deep Learning classification methods to distinguish between DS and MDR tuberculosis lesion appearance but no success was achieved there. Finally, since none of the additional approaches provided any significant increase of classification performance, only one run was submitted for subtask #1. In case of subtask #2, one single run was submitted as well.
In the final table of results the submitted run for subtask #1 was ranked as 7th among the 28 submitted runs with area under ROC-curve (AUC) equal to 0.5415 and prediction accuracy of 49.3% on the test image dataset. The best result in terms of AUC value was achieved by MedGIFT team [15] and resulted in AUC = 0.5825. Best drug resistance prediction accuracy of 56.8% was achieved by HHU DBS team [16].
For subtask #2, the submitted run was ranked 7th among the total number of 23 runs with recognition accuracy of 39.0% and Cohen's Kappa equal to 0.196. The best results in this subtask were obtained by SGEast team [17] which corresponded to Cohen's Kappa value of 0.2438 and recognition accuracy of 40.3%.
In this paper, image classification methods employed by UIIP group for Im-ageCLEF 2017 tuberculosis task were described. Being tested within the two subtasks, image description methods based on co-occurrence of voxels and supervoxels proved to be efficient for description of textural appearance of 3D medical images. It should be noticed that even the top-performing run submitted for subtask #1 resulted in AUC = 0.59 which is pretty low in terms of classification task. The conclusion which can be drawn is that the task of prediction of tuberculosis drug resistance based on a single CT image probably cannot be solved with a reasonable accuracy.
This study was supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project OISE-16-62631-1.