ImageCLEF 4 is the image retrieval task of CLEF (Conference and Labs of the Evaluation Forum). ImageCLEF was first held in 2003 and in 2004 a medical task was added that has been held every year since then [1][2][3][4]. More information on the other tasks organized in 2018 can be found in [5] and the past editions are described in [6][7][8][9].
Tuberculosis (TB) is a bacterial infection caused by a germ called Mycobacterium tuberculosis. About 130 years after its discovery, the disease remains a persistent threat and a leading cause of death worldwide [10]. This bacteria usually attacks the lungs, but it can also damage other parts of the body. Generally, TB can be cured with antibiotics. However, the greatest disaster that can happen to a patient with TB is that the organisms become resistant to two or more of the standard drugs. In contrast to drug sensitive (DS) TB, its multi-drug resistant (MDR) form is much more difficult and expensive to recover from. Thus, early detection of the MDR status is fundamental for an effective treatment. The most commonly used methods for MDR detection are either expensive or take too much time (up to several months) to really help in this scenario. Therefore, there is a need for quick and at the same time cheap methods of MDR detection. In 2017, ImageCLEF organized the first challenge based on Computed Tomography (CT) image analysis of TB patients [11], with a dedicated subtask for the detection of MDR cases. The classification of TB subtypes was also proposed in 2017. This is another important task for TB analysis since different types of TB should be treated in different ways. Both subtasks were also proposed in the 2018 edition where we extended their respective datasets. Moreover, a new subtask was added based on assessing a severity score of the disease given a CT image.
This article first describes the three tasks proposed around TB in 2018. Then, the datasets, evaluation methodology and participation are detailed. The results section describes the submitted runs and the results obtained for the three subtasks. A discussion and conclusion section ends the paper.
2 Tasks, Datasets, Evaluation, Participation
Three subtasks were organized in 2018. Two were common with the 2017 edition and one new subtask was added:
-Multi-Drug Resistance detection (MDR subtask); -Tuberculosis Type classification (TBT subtask); -Severity Scoring assessment (SVR subtask). This section gives an overview of each of the three subtasks.
Multi-drug Resistance Detection: As in 2017, the goal of the MDR subtask was to assess the probability of a TB patient having a resistant form of TB based on the analysis of a chest CT scan alone. The dataset for this subtask was increased from the previous year but the subtask remained as a binary classification problem even though several levels of resistances exist.
Tuberculosis Type Classification: This subtask is also common with the 2017 edition and, like in the MDR subtask, we increased the dataset. The goal of the TBT subtask is to automatically categorize each TB case into one of the following five TB types: Infiltrative, Focal, Tuberculoma, Miliary, and Fibrocavernous. The distribution of cases among the classes is not balanced but the distributions are similar in the training and the test data.
Severity Scoring: This subtask aims at assessing a TB severity score based only on a chest CT image. The severity score is a cumulative score of severity of a TB case assigned by a medical doctor. Originally, the score varied from 1 ("critical/very bad") to 5 ("very good"). In the process of scoring, the medical doctors considered many factors like pattern of the lesions, results of microbiological tests, duration of treatment, patient age and other criteria.
For each of the three subtasks, a separate dataset was provided, all containing 3D CT images stored in the NIfTI (Neuroimaging Informatics Technology Initiative) file format with slice resolution of 512×512 pixels and a number of slices varying from about 50 to 400. A set of relevant meta-data such as age and gender was provided for each subtask. The entire dataset including CT images and associated meta-data were provided by the Republican Research and Practical Center for Pulmonology and Tuberculosis that is located in Minsk, Belarus. The data were collected in the framework of several projects that aim at the creation of information resources on lung TB and drug resistance challenges. The projects were conducted by a multi-disciplinary team and funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH), U.S. Department of Health and Human Services, USA, through the Civilian Research and Development Foundation (CRDF). The dedicated web-portal5 developed in the framework of the projects stores information of more than 940 TB patients from five countries: Azerbaijan, Belarus, Georgia, Moldova and Romania. The information includes CT scans, X-ray images, genome data, clinical and social data.
In the framework of the ImageCLEF 2018 TB task, automatically extracted masks of the lungs were provided for all CT images. These masks were extracted using the method described in [12]. The segmentations were analyzed based on the number of lungs found and the size ratio of the lungs in a supervised manner. Only those segmentations with anomalies on these two metrics were visualized and evaluated accordingly. A total of 32 images out of 2,287 presented a problematic mask, 8 including areas outside the lungs and 24 containing only one lung. The 8 inaccurate masks were corrected by fusing the above mentioned method and the registration-based segmentation used in [13]. The other 24 masks (20 from the TBT subtask and 4 from the MDR subtask) could not be properly labeled due to the size and/or damage of one lung. In these cases, the masks provided to the participants only contained one label (right lung).
The dataset for this task is an extension of the one used in the 2017 edition. Particularly, the training and test sets of this subtask were extended by adding patients with extensively drug-resistant (XDR) TB, which is a rare and more severe subtype of MDR TB. Along with the 3D CT images and lung masks, the age and gender of each patient were provided. The dataset includes only HIV-negative patients with no relapses. Each patient was classified into one the two classes: drug sensitive (DS) or multi-drug resistant (MDR). A patient was considered DS if the TB bacteria was sensitive to all the anti-tuberculosis drugs tested. All XDR patients were considered to belong to the MDR class. Table 1 contains the number of patients in each set. Tuberculosis Type Classification The dataset used in this subtask includes chest CT scans of TB patients along with the TB type and patient age at the moment of the scan. Like the MDR dataset, the TBT 2017 dataset was extended for the 2018 edition. In this case, new CT scans of the same patients involved in 2017 were added and also some CT images of new patients. In the TBT 2018 dataset, for each patient there are between 1 and 9 CT scans acquired at different time points. All scans of the same patient were diagnosed with the same TB type by expert radiologists. Figure 1 shows one example for each of the five TB types. Moreover, Figure 2 shows examples of two patients with three CT scans each. The CT slices in both figures are shown using a Hounsfield Unit (HU) window with center at -500 HU and width of 1400 HU. The number of CT scans and patients in each TB type set are shown in Table 2.
The data for the SVR subtask includes 279 CT scans with known TB severity scores ranging from 1 to 5 assigned by medical doctors. Each CT scan corresponds to a specific TB patient. To treat this subtask as a binary classification problem, the severity scores were grouped so that values 1, 2 and 3 corresponded to "high severity" class, and values 4 and 5 corresponded to "low severity". Table 3 contains the number of patients of each severity class in the sets.
Miliary Fibro-cavernous
Similar to 2017, the participants were allowed to submit up to 10 runs to each of the three TB subtasks. In the case of the MDR subtask, the participants had to provide the probability for the TB cases to belong to the MDR class ranging from 0 to 1. These probabilities were used to build Receiver Operating Characteristic (ROC) curves. Since the MDR dataset was not perfectly balanced and had a relatively small size, Area Under the ROC Curve (AUC) was used to evaluate the participant runs. We provided the accuracy of the binary classification using a standard threshold of 0.50.
In the case of the TBT task, the participants had to predict the TB type of each patient, and submit a run containing a category label in the set {1, 2, 3, 4, 5}. Considering that a high number of patients in the dataset had multiple CT scans with the same TB type, the evaluation was performed patient-wise. Cohen's Kappa coefficient was provided for each run along with the 5-class prediction accuracy. Cohen's Kappa is not sensitive to unbalanced datasets, which is the case for the data used here.
The runs submitted for the severity scoring subtask were evaluated in two ways. One used the original severity scores from 1 to 5 and the task was to predict those numerical scores as precise as possible (a regression problem).
Here, Root Mean Square Error (RMSE) was computed between the ground truth severity and the predicted scores provided by the participants. Alternatively, the original severity score was transformed into two classes, where scores from 1 to 3 corresponded to "high severity" and the 4 and 5 scores corresponded to the "low severity" class. In this case the participants had to provide the probability of TB cases to belong to the "high severity" class. The corresponding results were evaluated using AUC.
In 2018 there were 85 registered teams and 33 signed the end user agreement. Finally, 11 groups from 9 countries participated in one or more subtasks and submitted results. These numbers are similar to 2017, where there were 94 registered teams, 48 that signed the end user agreement, and 9 teams from 9 countries submitting results. Table 4 shows the list of participants and the subtasks where they participated. One of the groups (HHU-DBS) participated in two subtasks with different approaches developed by a different set of authors. Therefore, their approaches are referred as HHU-DBS 1 and HHU-DBS 2 in the following sections.
This section provides the results obtained by the participants in each of the subtasks.
Table 5 shows the results obtained for the MDR detection subtask. The runs were evaluated using ROC curves produced from the probabilities provided by the participants. The results in the table are sorted by AUC in descending order. The accuracy is given in the table as well. Additionally, Figure 3 shows the highest AUC values achieved by the participants compared to the best result obtained in the 2017 edition. It is worth to notice that the image-based detection task of MDR TB remains very challenging and so far has no solution with a sufficiently high prediction accuracy for being useful in clinical practice. Recent articles report the presence of statistically significant links between drug resistance and multiple thick-walled caverns [14]. However, computerized methods show a performance of image-based MDR TB detection barely beyond the level of statistical significance compared to a random classifier [6,15,16].
The best result in terms of AUC was achieved by VISTA@UEvora team with an AUC of 0.6178 [17]. The team used conventional approaches for the extraction of quantitative image descriptors, such as statistical moments, fractal dimension, gray-level co-occurrence matrices and their derivative features. A set of conventional classification methods was used for prediction in all the three subtasks. Their best run in terms of classification accuracy (0.5763) ranked 3rd place among the participant runs and is not the same run that had the best AUC. The second highest AUC of 0.6114 was achieved by the San Diego VA HCS/UCSD [18] with an approach based on splitting the 3D CT scans into a set of 2D images and using a pre-trained ResNeXt deep network for classification. This run achieved the highest MDR detection accuracy (0.6144). The third highest AUC was obtained by HHU-DBS 1 [19]. They used 3D deep Convolutional Neural Networks (CNNs) combined with decision trees and obtained 0.5810 AUC and 0.5720 classification accuracy with their best run. The UniversityAlicante group used two approaches: one based on 2D CNNs and the other based on Optical Flow (OF) [20]. The best AUC among this group's runs was obtained using only patient age and gender information and ranked 10th among all participant runs with a AUC of 0.5669. Other runs obtained lower AUC. This OF-based approach for CT image analysis resulted in an accuracy of 0.5339 and ranked 20th. The single run submitted by the UIIP BioMed group ranked 14th in AUC and 36th in accuracy with an AUC of 0.5558 and an accuracy of 0.4576 [21]. A technique for automatic detection of lesions of different types in a six-region division of the CT lung volume was used. A separate dataset with labeled lesions in CT was used for training the lesion detection algorithm. A Random Forest (RF) classifier was used for the prediction of the final classes and scores in all three subtasks. Methods based on a graph-model of the lungs and 3D texture analysis were used by MedGIFT group [22]. Their best runs resulted in the 22nd highest AUC (0.5237) and the 2nd highest accuracy (0.5932). Finally, the LIST group used a hybrid approach that combined 3D CNNs with linear SVM classifiers for MDR detection and TB type classification [23]. The single run submitted by the group obtained an AUC of 0.5029 and an accuracy of 0.4576 and ranked 28th and 37th, respectively. The information about age and gender of TB patients was used only by two participating groups: HHU-DBS 1 and UIIP BioMed.
Table 6 shows the results obtained for the TBT subtask. The runs were evaluated on the test set of images using the unweighted Cohen Kappa coefficient and overall classification accuracy. The results are sorted by Cohen's Kappa in descending order. Figure 4 shows the highest Kappa values achieved by the participants. The true positive rates of the different TB types are shown in Figure 5.
In the TBT subtask, most of the teams used the same methods as they used for the MDR detection. The best result in terms of both Kappa and classification accuracy was achieved by the UIIP BioMed group with the use of a lesionbased TB descriptor and a RF classifier. The run resulted in a Kappa of 0.2312 and a classification accuracy of 0.4227. Instead of using all the available CT series, this group only used the first scan of a patient for the classification of the TB type. The second highest Kappa was achieved by the fau ml4cv group that participated only in the TBT subtask [24]. An ensemble of 3D CNNs was used, achieving a Kappa of 0.1736 and an accuracy of 0.3533 with their best run. The graph-based approach of the MedGIFT team resulted in the 2nd best classification accuracy (0.3849) and the 3rd highest Kappa (0.1706). The best runs of VISTA@UEvora, San Diego VA HCS/UCSD, UniversityAlicante and LIST resulted in Kappa values of 0.1664, 0.1474, 0.0204, and -0.0024 respectively. The MostaganemFSEI group participated in the TBT classification and the SVR subtasks. The algorithm employed by them was based on splitting the 3D CT scans into 2D slices, extracting semantic descriptors using a trained CNN and applying conventional classification methods [25]. They obtained a Kappa of 0.0629 and an accuracy of 0.2744. It is worth to highlight that only the fau ml4cv and San Diego VA HCS/UCSD groups obtained a true positive rate higher than a random classifier in all five TB types (see Figure 5).
The results obtained for the severity scoring subtask are shown in Table 7. The best RMSE achieved by the participating groups and the corresponding AUCs are shown in Figures 6 and 7. The best results in terms of regression were obtained by the UIIP BioMed group with an RMSE of 0.7840, which also achieved the 6th best classification result with an AUC of 0.7025. The highest classification result was achieved by the MedGIFT group with an AUC of 0.7708. The MedGIFT group's best regression obtained an RMSE of 0.8513, which is the second best result. The third best RMSE (0.8883) was obtained by the VISTA@UEvora group. The same run ranked on the 21st place for classification with an AUC of 0.6239. The third best result for classification was obtained by the San Diego VA HCS/UCSD group with an AUC of 0.6984, which corresponds to the 7th best result. Their best regression is an RMSE of 1.2153, which is at rank 30. The HHU-DBS 2 team used a feature-based approach for scoring the severity of TB based on a set of conventional methods [26]. The approach employed image binarization and extraction of features including the presence of calcifications, lung wateriness, cavities, infection ratio, HU histograms and lung shape to characterize the volumes. The group obtained the 10th best RMSE (0.9626) and 8th best AUC (0.6862). The MostaganemFSEI group achieved an RMSE of 0.9721 and an AUC of 0.6127. Middlesex University participated only in the SVR subtask. The group employed an approach based on using deep residual learning, training on a set of overlapping 128 × 128× depth blocks, assessing the TB severity for each block and gathering the results [27]. This allowed to achieve an RMSE of 1.0921 and an AUC of 0.6534 that correspond to the 24th and 14th positions. It is important to highlight that all groups obtained an AUC higher than a random classifier (AUC of 0.50) with all their runs.
Similar to 2017, the results obtained by the participants in the MDR detection subtask demonstrate that the task of a fully automatic image-based detection of drug resistance is extremely difficult. Despite the addition of XDR TB cases into the dataset and the inclusion of information about patient age and gender, the MDR detection performance still remains at a level relatively close to a random classification with the highest reached AUC of 0.6178 and a 61.4% prediction accuracy. The overall increase of prediction performance with respect to the 2017 edition might be caused by the addition of more severe cases with XDR TB into the dataset. Using information about patient age and gender could also improve the MDR detection results as suggested by the baseline submitted by UniversityAlicante group [20].
In the second subtask, the overall results of TB type classification are slightly worse than in 2017. This might be caused by the decreased balance of TB classes in the dataset. Using more than one CT scan per patient could also confuse prediction methods and worsen the final results. However there is a certain improvement in prediction of class T2 (Focal TB) demonstrated by most of the participants' results.
The results of SVR subtask are encouraging, since the actual assessment of the TB severity score is done using various clinical information sources, not only CT image data. Most of the results achieved by the participants obtained a RMSE of the severity score below 1 in a 5-grade scoring system. The best results obtained using only CT volumes are close to the results reported in [28], where the authors used clinical and laboratory data including drug resistance, presence of TB symptoms, etc. in addition to the images. Extension of the dataset and usage of clinical and laboratory data is expected to improve the severity scoring results.
Overall, the 2018 edition of the ImageCLEF TB task showed an improvement with respect to the 2017 edition in terms of number of participants, data provided, results obtained and the variety of methods proposed. This shows a high interest in this topic and also the importance of the data that were generated.
This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030-146804) and by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-17-63599-1 "Year 6: Belarus TB Database and TB Portals".