In 2018, ImageCLEF proposed a task using CT (Computed Tomography) scans of patients with tuberculosis (TB). The task was divided into three subtasks: multi{drug resistance detection, TB type classi cation, and severity scoring. In this work we present a graph model of the lungs capable of characterizing TB patients with di erent lung problems. The graph contains a xed number of nodes with weighted edges based on dissimilarity measures between texture descriptors computed in the nodes. This model encodes the texture distribution along the lungs, making it suitable for describing patients with di erent TB types. The results show the strength of the technique, leading to high results in the three subtasks.
ImageCLEF (the image retrieval and analysis evaluation campaign of the Cross{
Language Evaluation Forum, CLEF) has organized challenges on image
classication and retrieval since 2003 [
When tuberculosis a ects the lungs, several visual patterns can be seen in a CT image. These patterns are usually characteristic of the underlaying TB type. Moreover, their spread into parts of the lung is a good indicator of the severity of the diseases. However, the nal diagnosis usually required other analyses than only the images [10]. Our approach is able to obtain a global texture{based description of the lungs. It consists of creating a graph model of the lungs where the nodes represent lung regions and the edges encode relations between the texture inside the regions.
The following section contains a brief overview of the subtasks and datasets of the ImageCLEF 2018 TB task. More detailed information on the task can be found in the overview article [7]. Section 3 explains the process of building the texture{based graph model of the lungs and all the variations tested for this task in detail. The results obtained by this approach in the three subtasks are shown in Section 4. Finally, Section 5 concludes our participation in this challenge. 2
The ImageCLEF 2018 TB task proposed three subtasks: i) Multi{drug resistance (MDR) detection, ii) Tuberculosis type (TBT) classi cation, iii) and severity scoring (SVR). For the three subtasks, volumetric chest CT images and automatic segmentations of the lungs were provided by the organization. No other lung segmentation was attempted in this work and the masks provided were used. These masks were obtained with the method described in [11].
The challenge was divided into two phases. In the rst phase, the organizers released for each subtask a set of patient CT volumes as training set along with their lung masks, groundtruth labels and meta{data including age and gender of the patients. In the second phase, the test set with the corresponding lung segmentations and meta{data were provided. The groundtruth for the test data was never released. In our approach we only used the CT images and no meta{ data. The evaluation on the test set was performed by the organizers after the scheduled deadline for all runs that were submitted in time. The number of CT volumes for each subtask are speci ed in Tables 1, 2, and 3. This section details the process of building our graph{based model, the extraction of a descriptor vector from the CT images, and the classi cation algorithm applied. The same technique was applied for describing the patients of the three subtasks. Our method consists of creating a graph model of the lungs, with nodes based on a geometrical atlas and weighted edges encoding dissimilarities between 3D texture descriptors of each atlas region. Figure 1 shows the pipeline to obtain the graph model. Our approach is based on 3D texture features that require having isometric voxels. We rst made the 3D images and the lung masks isometric. After analyzing the multiple resolutions and the inter{slice distances in the dataset, we opted for a voxel size of 1 mm to capture a maximum of the available information. To build a graph with a xed structure over the lung anatomy we opted for a geometric division of the lung witha xed number of regions. We chose the atlas developed by Depeursinge et al. in [12]. This atlas is only based on the mask of the lungs and provides 36 geometric regions dividing the lungs as shown in Figure 1. Two state{of{the{art 3D texture features were selected to describe the texture inside the lungs. The rst method is a histogram of gradients based on the Fourier transform HOG (FHOG) introduced in [13]. We used 28 3D directions for the histogram obtaining a 28{dimensional feature vector per image voxel (fH 2 R28). The second approach is the locally{oriented 3D Riesz-wavelet transform introduced by Dicente et al. in [14] and based on [15]. The parameters that obtained the best results in the above mentioned article were used in our approach. These are: 3rd{order Riesz transform, 4 scales and 1st{order alignment. This con guration resulted in 40{dimensional feature vectors for each image voxel. The feature vector for a single voxel was then reduced to be 10{dimensional containing the energy of each lter along the 4 scales (fR 2 R10).
Feature Vector of a Region: Several feature vectors can be extracted from a region r using the above mentioned texture descriptors fH and fR. Given a region ri, we extracted the mean ( i) and standard deviation ( i) of the features inside the region, i.e.: i(fH ), i(fH ), i(fR), and i(fR). Hence, four feature vectors were obtained for each atlas region. 3.4
We used a weighted undirected graph G(N ; E ) with 36 nodes and 84 edges to model the lung. Each node Ni 2 N corresponds to a region ri in the geometric atlas. The edges were based on the region adjacency de ned by the atlas. In this case, there is an edge Ei;j between nodes Ni and Nj if regions ri and rj are neighbors in the atlas, i.e., if they are 3D adjacent. Moreover, it has 18 additional edges connecting each pair of nodes representing opposite regions inside the atlas with respect to the left/right division of the lungs. This con guration resulted in 84 edges and is shown in Figure 1.
The weight wi;j of an edge Ei;j was de ned using two dissimilarity measures between the regional features: the Euclidean and the correlation distances. Depending on the measure selected, a di erent graph model was obtained. 3.5
The feature descriptor wp of a patient p was de ned as the ordered list of weights wi;j 2 Gp. Since wi;j = wj;i, this descriptor was 84{dimensional for all patients. These vectors were normalized using a Z{score normalization based on the patients of the training set. The elements composing this vector can not be seen independently since they encode the structure of the graph. In this case, the normalization was applied to all the components simultaneously, i.e., the mean and variance needed for the Z{score normalization were computed over all edge weights from the training patients.
Patient Descriptor Concatenation: For a given distance between regional features (corr or euc), four normalized patient descriptor vectors wp were obtained. These were: w H , w H , w R , and w R . We ran a di erent experiment for each of these patient descriptors. Moreover, we also tested ve concatenations of these descriptors in order to better describe each patient, resulting in nine experiments. The tested concatenations were de ned as: { Mean and std of FHOG: w^ = (w H jjw H ). { Mean and std of Riesz: w^ = (w R jjw R ). {{ SMtedaonf oFfHFOHGOGanadnRdiResize:szw^: w=^ =(w(wH jHjwjjwR )R. { Mean and std of FHOG and Riesz: w^ = (w ).
H jjw H jjw
R jjw
R ).
Feature Space Reduction: For some of the concatenations of patient descriptors, the feature space dimension was signi cantly larger than the number of patients in a single class. To avoid the known problems of using such large feature spaces, we performed feature space reduction by selecting the dimensions with higher correlation with respect to the groundtruth labels. This technique reduced the size of the feature space by two approximately. The nal experiments were performed both using this feature space reduction and with no reduction. Tested runs: We performed 36 experiments (or runs) in total per subtask, generated by the di erent options explained. Table 4 summarizes all possible options for each step. 3.6
Multi{class support vector machine (SVM) classi ers with RBF kernel were used for each run in the three subtasks, particularly, 2{class for the MDR subtask, and 5{class for the TBT and SVR subtasks. Grid search over the RBF parameters cost C and gamma was applied. Since the data were normalized, both C and moved in f2 10; 2 9; : : : ; 210g. The best C and combination for a run was set as the one with highest cross{validation accuracy in the training set of each subtask.
MDR subtask: The submission run for the this subtask had to contain the probability for a patient of being multi{drug resistant. In this case, we used the probabilities from the SVM algorithm.
TBT subtask: For this subtask, the dataset contained more than one CT scan per patient in most of the cases. However, the classi cation had to be performed at the patient level. In this case, we averaged the probabilities provided by the SVM algorithm over all the images of the same patient. The overall most probable class was assigned to the patient.
SVR subtask: In this subtask the submission le had to contain the severity score (1 to 5) and the probability of belonging to the high severity class for each patient. We considered the severity score as a class in the SVM algorithm, assigning the most probable class as the predicted severity score. According to the organizers, the high class corresponded to a severity score in [1; 3]. In this case, for each patient, we obtained the probability of class high by adding the SVM probabilities of belonging to classes 1 to 3. 4
A total of 10 runs could be submitted in each ImageCLEF 2018 TB subtask. We chose them based on the performance obtained in the training set. For this, we used the same indicators as the ones used by the organizers. These were: Accuracy (Acc) and Area Under the Curve (AUC) for the MDR subtask, Acc and unweighted Cohen's Kappa (Kappa) for the TBT subtask, and AUC and Root Mean Square Error (RMSE) for the SVR task. Additionally, we base our selection of the best runs on the cross{validation accuracy extracted from the SVM classi er during the training phase. Tables 5, 6, and 7 show the results obtained by each submitted run along with the best result obtained by other participants with respect to each performance measure. All results were provided by the organizers of the task. Our group participated as the MedGIFT group. In the case of the MDR subtask, only 9 runs were evaluated since one submission failed. UIIP BioMed Not applicable 0.2312 MedGIFT FHOG & Riesz / std / euc / mostCorr 0.1706 MedGIFT Riesz / mean & std / euc / mostCorr 0.1674 MedGIFT FHOG & Riesz / mean & std / corr / mostCorr 0.1531 MedGIFT FHOG & Riesz / mean / euc / none 0.1517 MedGIFT Riesz / std / euc / mostCorr 0.1494 MedGIFT FHOG & Riesz / mean & std / corr / none 0.1356 MedGIFT FHOG / mean & std / euc / mostCorr 0.0949 MedGIFT FHOG / std / corr / none 0.0855 MedGIFT FHOG & Riesz / std / corr / mostCorr 0.0787 MedGIFT FHOG / std / corr / mostCorr 0.0589
Considering the accuracy, the proposed method ranked 2nd in the MDR and TBT subtasks. However, considering the AUC in the MDR subtask, the best result achieved ranked 22nd. In the case of the TBT subtask, our best run with respect to the accuracy is also the best with respect to the Kappa measure, obtaining the 3rd place. For the SVR subtask, our method ranked 1st according to the AUC and 2nd considering the RMSE. Overall, the results are good when compared with other participants. However, they are still far from perfect in the three subtasks because for real clinical applications a higher accuracy seems necessary. Moreover, they re ect that the optimization of the SVM parameters was based on the cross{validation accuracy. This work presents an updated version of our previously developed graph model of the lung based on regional 3D texture features for describing lungs a ected by tuberculosis. The participation in the ImageCLEF 2018 TB task allows for an objective comparison between methods since the evaluation was performed by the organizers. The results proved the suitability of our approach for detecting, classifying and scoring patients with TB. However, the results show that there is still room for improvement, particularly in the MDR subtask were results were relatively close to random for all participants. We believe that the results could have been better if the optimization of the classi er parameters would have been done based on other performance measures rather than only on the SVM cross{validation accuracy.
This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030{146804). overview of ImageCLEF at the CLEF 2015 labs. In: Working Notes of CLEF 2015.
Lecture Notes in Computer Science. Springer International Publishing (2015) 4. Markonis, D., Holzer, M., Dungs, S., Vargas, A., Langs, G., Kriewel, S., Muller, H.: A survey on visual information search behavior and requirements of radiologists.
Methods of Information in Medicine 51(6) (2012) 539{548 5. Muller, H., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S.: Creating a classi cation of image types in the medical literature for visual categorization. In: SPIE Medical Imaging. (2012) 6. Ionescu, B., Muller, H., Villegas, M., de Herrera, A.G.S., Eickho , C., Andrearczyk, V., Cid, Y.D., Liauchuk, V., Kovalev, V., Hasan, S.A., Ling, Y., Farri, O., Liu, J., Lungren, M., Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., Gurrin, C.: Overview of ImageCLEF 2018: Challenges, datasets and evaluation. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018), Avignon, France, LNCS Lecture Notes in Computer Science, Springer (September 10-14 2018) 7. Dicente Cid, Y., , Liauchuk, V., Kovalev, V., , Muller, H.: Overview of ImageCLEFtuberculosis 2018 - detecting multi-drug resistance, classifying tuberculosis type, and assessing severity score. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, CEUR-WS.org <http://ceur-ws.org> (September 10-14 2018) 8. Dicente Cid, Y., Kalinovsky, A., Liauchuk, V., Kovalev, V., , Muller, H.: Overview of ImageCLEFtuberculosis 2017 - predicting tuberculosis type and drug resistances. In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceur-ws.org> (September 11-14 2017) 9. Dicente Cid, Y., Batmanghelich, K., Muller, H.: Textured graph-model of the lungs for tuberculosis type classi cation and drug resistance prediction: participation in ImageCLEF 2017. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceur-ws.org> (September 11-14 2017) 10. Jeong, Y.J., Lee, K.S.: Pulmonary tuberculosis: up-to-date imaging and management. American Journal of Roentgenology 191(3) (2008) 834{844 11. Dicente Cid, Y., Jimenez-del-Toro, O., Depeursinge, A., Muller, H.: E cient and fully automatic segmentation of the lungs in CT volumes. In Orcun Goksel, Jimenez-del-Toro, O., Foncubierta-Rodriguez, A., Muller, H., eds.: Proceedings of the VISCERAL Challenge at ISBI. Number 1390 in CEUR Workshop Proceedings (Apr 2015) 31{35 12. Depeursinge, A., Zrimec, T., Busayarat, S., Muller, H.: 3D lung image retrieval using localized features. In: Medical Imaging 2011: Computer{Aided Diagnosis.
Volume 7963., SPIE (2011) 79632E 13. Liu, K., Skibbe, H., Schmidt, T., Blein, T., Palme, K., Brox, T., Ronneberger, O.: Rotation-invariant hog descriptors using fourier analysis in polar and spherical coordinates. International Journal of Computer Vision 106(3) (2014) 342{364 14. Dicente Cid, Y., Muller, H., Platon, A., Poletti, P.A., Depeursinge, A.: 3{D solid texture classi cation using locally{oriented wavelet transforms. IEEE Transactions on Image Processing 26(4) (April 2017) 1899{1910 15. Depeursinge, A., Foncubierta-Rodr guez, A., Van De Ville, D., Muller, H.: Rotation{covariant texture learning using steerable Riesz wavelets. IEEE Transactions on Image Processing 23(2) (February 2014) 898{908