ImageCLEF 4 is the image retrieval task of CLEF (The Cross Language Evaluation Forum). ImageCLEF was first held in 2003 and since 2004 a medical task has been added [1,2]. More information on the other tasks organized in 2017 can be found in [3] and the past editions are described in [4][5][6].
About 130 years after the discovery of Mycobacterium tuberculosis, the disease remains a persistent threat and a leading cause of death worldwide. The greatest disaster that can happen to a patient with tuberculosis (TB) is that the organisms become resistant to two or more of the standard drugs. In contrast to drug sensitive (DS) tuberculosis, its multi-drug resistant (MDR) form is much more difficult and expensive to recover from. Thus, early detection of the drug resistance (DR) status is of great importance for effective treatment. The most commonly used methods of DR detection are either expensive or take too much time (up to several months). Therefore there is a need for quick and at the same time cheap methods of DR detection. One of the possible approaches for this task is based on Computed Tomography (CT) image analysis, where many details can be seen. In many countries X-ray imaging is used for disease detection and CT images are only taken when required. The detection of TB subtypes is another important task for tuberculosis analysis, as different types of tuberculosis can be treated in different ways.
This article first describes the two tasks proposed around tuberculosis in 2017. Then, the data sets, evaluation methodology and participation are detailed. The results describe the submitted runs and results obtained for the two sub tasks and a discussion and conclusion ends the paper.
2 Tasks, Data Sets, Evaluation, Participation
Two subtasks were organized in 2017:
multi-drug resistance detection (MDR task); tuberculosis type classification (TBT task). This section gives an overview of each of the two subtasks.
The goal of the MDR subtask is to assess the probability of a TB patient having resistant form of tuberculosis based on the analysis of a chest CT scan. The training data were given as binary cases even though several levels of resistances exist.
The goal of the tuberculosis type subtask is to automatically categorize each TB case into one of the following five types: Infiltrative, Focal, Tuberculoma, Miliary, and Fibro-cavernous. The distribution of cases among the classes is not fully equal but distributions are similar in the training and the testing data.
For both subtasks 3D CT images were provided with slice size of 512×512 pixels and number of slices varying from about 50 to 400. All images are part of a tuberculosis screen program in Belarus, which has a high percentage of tuberculosis cases compared to other countries. All the CT images were stored in NIFTI file format with .nii.gz file extension (g-zipped .nii files). This file format stores raw voxel intensities in Hounsfield units (HU) as well as the corresponding image metadata such as image dimensions, voxel size in physical units, slice thickness, etc.
All lung tuberculosis data including CT images and associated annotations were provided by Republican Research and Practical Center for Pulmonology and Tuberculosis which is located in Minsk, Belarus. The data were collected in framework of several projects aimed at creation of information resources on lung tuberculosis and drug resistance problem. The projects were conducted by a multi-disciplinary team and funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH), U.S. Department of Health and Human Services, USA through the Civilian Research and Development Foundation (CRDF). The dedicated web-portal5 developed in framework of the projects stores information on more than 940 tuberculosis patients from five countries: Azerbaijan, Belarus, Georgia, Moldova and Romania. The information includes CT scans, X-ray images, genome data, clinical and social data.
Moreover, for all patients in both subtasks we provided automatically extracted masks of the lungs to make further processing easier. These masks were extracted using the method described in [7]. Figure 1 shows one CT example with the provided mask as overlay. Multi-drug Resistance Detection For this subtask, a dataset of 3D CT images was used along with a set of clinically relevant metadata. The dataset includes only HIV-negative patients with no relapses. Each patient was classified into one the two classes: drug sensitive (DS) or multi-drug resistant (MDR) cases.
A patient was considered DS if tuberculosis bacteria were sensitive to all the antituberculosis drugs tested. In case of MDR tuberculosis the corresponding drug susceptibility tests showed resistance to at least Isoniazid and Rifampicin -the two most powerful anti-TB drugs. In reality there are several levels of resistances but to make the task as easy as possible only these two most representative classes were retained. Tuberculosis Type Classification The dataset used in this subtask includes chest CT scans of TB patients along with the TB type. Figure 2 shows one example for each of the five TB types. Total patients 500 300
The participants were allowed to submit up to 10 runs to the tuberculosis task for each of the two sub tasks. In the case of the MDR tasks, participants had to provide, for each patient in the test set, the probability between 0 and 1 of belonging to the MDR class. These probabilities were used to build Receiver Operating Characteristic (ROC) curves. Since the MDR dataset was not perfectly balanced and had a relative small size, we decided to use the Area Under the ROC Curve (AUC) to rank the participant runs. Moreover, we provided the accuracy of the binary classification using a standard threshold of 0.50. In the case of the TBT task, the participants had to predict the TB type of each patient, and submit a run containing a categorical label in the set {1, 2, 3, 4, 5}. For this task Cohen's Kappa was provided for each run along with the weighted average accuracy along the classes. The former measure is not sensitive to unbalanced datasets as in this task.
94 teams registered for the tuberculosis task and 48 signed the end user agreement. Finally, 9 teams from 9 countries submitted a run for at least one of the two subtasks. Table 3 shows the list of participants and the task(s) where they participated.
This section provides the results obtained by the participants in each of the subtasks.
Table 4 shows the results obtained for the MDR detection subtask. The runs were evaluated on the test set of images using ROC curves produced from the probabilities provided by participants. The results in the table are sorted by AUC in descending order. Figure 3 shows the ROC curve for the best run of each participant. Moreover, the best operating point in the curve is highlighted. The accuracy is given in the table as well. It should be noticed that the task of image-based recognition of MDR for tuberculosis is very difficult and challenging. It was not clear from the start whether this task is actually possible with visual data alone. There are several studies dedicated to this problem that reported presence of statistically significant links between drug resistance status and various features of the patient lungs [8][9][10][11]. However, none of the studies reported any precise classification of drug resistance with accuracy far beyond the level of statistical significance.
Among the submitted runs, the three best results were achieved by the MedGIFT team. Methods based on a graph-model of the lungs were used for the description of lung CT appearance [12]. For different variations of the method the AUC calculated over the test set varied from 0.5112 to 0.5825. The SGEast team employed deep learning in both subtasks. Two-dimensional convolutional neural networks were used for describing the CT scans and recurrent neural networks were used for image classification [13]. Combined with preprocessing procedures such as image slicing and data augmentation the method resulted in 0.5620 AUC value for their best run. The UIIP team used an image description method based on the calculation of co-occurrence matrices of 3D supervoxels which allowed to achieve an AUC of 0.5415 [14]. The best performing result in terms of classification accuracy was achieved by the HHU DBS team by using deep convolutional networks which operate directly in 3D [15]. The method demonstrated a classification accuracy of 56.8% with an AUC of 0.5297 for the best run. The Batmanlab used a 3D intensity-based supervoxel image representation and various texture features in each supervoxel and achieved their maximum performance at 0.5241 AUC [16]. 2D deep learning techniques were used by MEDGIFT UPB along with CT image slicing and data augmentation, which resulted in an AUC of 0.5184 for their best run [17]. The Aegean Tubercoliosis group obtained an AUC of 0.4833 with their single run. However, no details on their technique were provided. Finally, the Bioinformatics UA team also used 3D convolutional neural networks in their study and achieved 46.0% of classification accuracy with an AUC of 0.4648 in the MDR subtask [18]. The authors believe that using data augmentation and weight regularization may potentially improve the results. coefficient in descending order. In addition, Figure 4 contains the confusion matrices of each team's best run. These matrices show the percentage of patients from each class classified in each TB type. When analyzing these matrices, we found that the best performance per TB type is found in the first 4 teams (SGEast, MEDGIFT UPB, Image Processing, and UIIP). These methods seem to be complementary, being expert each one in classifying a different TB type. Thus, the organizers of the task have performed a late fusion with the best run from these 4 best teams. The results of the corresponding fusion are included at the end of both Table 5 and Figure 4. In the tuberculosis type subtask, most of the teams used the same set of methods as they used for MDR detection subtask. The best results were achieved by the SGEast and MEDGIFT UPB groups with Cohen's Kappa of 0.2438 and 0.2329 respectively for their best runs. The Image Processing group used a technique based on splitting CT images into sets of small 2D patches followed by classification of image patches via convolutional neural networks [19]. The corresponding submitted run scored 0.2187 for Cohen's Kappa. The UIIP team used a different approach for TB type classification that uses extended multi-sort co-occurrence matrices of voxels in 3D images.
The only submitted run resulted in a Kappa score of 0.1956. The best runs of MedGIFT, BatmanLab and BioinformaticsUA groups achieved Cohen's Kappa values of 0.1623, 0.1533, and 0.0222, respectively. Finally, the fusion of the best runs of the 4 best teams achieved a 0.3263 Cohen's Kappa value. This result is an important improvement with respect the best participant run and shows how complementary the approaches of these four participants are. However, when analyzing its confusion matrix in Figure 4, this fusion method mainly improved the performance for T1. In all the other classes, the performance is below the best performance achieved in the corresponding class by at least one of the fused methods. In any case, the average accuracy obtained it is also superior (0.4867) to all submitted runs by an important margin.
The results obtained by the participants confirm the difficulty of the MDR task. Independently of the technique applied, all runs remained relatively close to the performance of a random classifier, meaning that there is likely a high potential for improvements. One explanation is that it is not possible to detect MDR patients based on images alone, and maybe additional data on the patients need to be combined with visual information. The results above random encourage to find and add more cases to this dataset and continue exploring the detection of drug resistance based on images, as a much larger training data set can maybe better cover the different subtypes, as can potentially the addition of multimodal information. On the other hand, there are other existing forms of TB drug resistance which are not considered with this study. For example, quick detection of Extensively drug-resistant (XDR) tuberculosis is also of a great scientific and practical interest.
The TB subtask results are clearly based on the suitability of the imaging techniques for a specific task. The best runs seem to exploit quite different aspects and each of them is best for one specific class. This complementarity is highlighted by the very good results of a simple late fusion of the four best runs. However, there is still room for improvement, especially in the types Focal (T2), Tuberculoma (T3), Miliary (T4), and Fibro-cavernous (T5). For this particular task, deep learning methods worked better than other approaches, obtaining the 6 best results. Nonetheless, the late fusion performed by the organizers combining the prediction of three deep learning techniques and one based on 3D texture features surpassed the best participant run. This suggests that different TB types may need to be described by different approaches and such an approach can lead to optimal results. 2017 was the first year of the ImageCLEF TB task. The number of registered participants and the results obtained show the interest of the community in this task, and encourages the inclusion of the TB task in the upcoming editions of ImageCLEF. There is still much to be learned and a detailed analysis of the results should help with this.
This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030-146804) and by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project OISE-16-62631-1.