Overview of the ImageCLEF 2017 Tuberculosis Task | Predicting Tuberculosis Type and Drug Resistances

Abstract

ImageCLEF is the image retrieval task of the Conference and Labs of the Evaluation Forum (CLEF). ImageCLEF has historically focused on the multimodal and language-independent retrieval of images. Many tasks are related to image classi cation and the annotation of image data as well as the retrieval of images. The tuberculosis task was held for the rst time in 2017 and had a very encouraging participation with 9 groups submitting results to these very challenging tasks. Two tasks were proposed around tuberculosis: (1) the classi cation of the cases into ve types of tuberculosis and (2) the detection of drug resistances among tuberculosis cases. Many di erent techniques were used by the participants ranging from Deep Learning to graph-based approaches and best results were obtained by a large variety of approaches. The prediction of tuberculosis types had relatively good performance but the detection of drug resistances remained a very di cult task. More research into this seems necessary.

Keywords

• Tuberculosis
• Computed Tomography
• Image Classi cation
• Drug Resistance
• 3D Data Analysis

-

ImageCLEF4 is the image retrieval task of CLEF (The Cross Language Evaluation Forum). ImageCLEF was rst held in 2003 and since 2004 a medical task has been added [ 1,2 ]. More information on the other tasks organized in 2017 can be found in [ 3 ] and the past editions are described in [4{6].

About 130 years after the discovery of Mycobacterium tuberculosis, the disease remains a persistent threat and a leading cause of death worldwide. The greatest disaster that can happen to a patient with tuberculosis (TB) is that the organisms become resistant to two or more of the standard drugs. In contrast to drug sensitive (DS) tuberculosis, its multi-drug resistant (MDR) form is 4 http://www.imageclef.org/ much more di cult and expensive to recover from. Thus, early detection of the drug resistance (DR) status is of great importance for e ective treatment. The most commonly used methods of DR detection are either expensive or take too much time (up to several months). Therefore there is a need for quick and at the same time cheap methods of DR detection. One of the possible approaches for this task is based on Computed Tomography (CT) image analysis, where many details can be seen. In many countries X-ray imaging is used for disease detection and CT images are only taken when required. The detection of TB subtypes is another important task for tuberculosis analysis, as di erent types of tuberculosis can be treated in di erent ways.

This article rst describes the two tasks proposed around tuberculosis in 2017. Then, the data sets, evaluation methodology and participation are detailed. The results describe the submitted runs and results obtained for the two sub tasks and a discussion and conclusion ends the paper. 2 2.1

Tasks, Data Sets, Evaluation, Participation

The Tasks in 2017

Two subtasks were organized in 2017: { multi-drug resistance detection (MDR task); { tuberculosis type classi cation (TBT task).

This section gives an overview of each of the two subtasks.

Multi-drug resistance detection The goal of the MDR subtask is to assess the probability of a TB patient having resistant form of tuberculosis based on the analysis of a chest CT scan. The training data were given as binary cases even though several levels of resistances exist.

Tuberculosis type classi cation The goal of the tuberculosis type subtask is to automatically categorize each TB case into one of the following ve types: In ltrative, Focal, Tuberculoma, Miliary, and Fibro-cavernous. The distribution of cases among the classes is not fully equal but distributions are similar in the training and the testing data. 2.2

Datasets

For both subtasks 3D CT images were provided with slice size of 512 512 pixels and number of slices varying from about 50 to 400. All images are part of a tuberculosis screen program in Belarus, which has a high percentage of tuberculosis cases compared to other countries. All the CT images were stored in NIFTI le format with .nii.gz le extension (g-zipped .nii les). This le format stores raw voxel intensities in Houns eld units (HU) as well as the corresponding image metadata such as image dimensions, voxel size in physical units, slice thickness, etc.

All lung tuberculosis data including CT images and associated annotations were provided by Republican Research and Practical Center for Pulmonology and Tuberculosis which is located in Minsk, Belarus. The data were collected in framework of several projects aimed at creation of information resources on lung tuberculosis and drug resistance problem. The projects were conducted by a multi-disciplinary team and funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH), U.S. Department of Health and Human Services, USA through the Civilian Research and Development Foundation (CRDF). The dedicated web-portal5 developed in framework of the projects stores information on more than 940 tuberculosis patients from ve countries: Azerbaijan, Belarus, Georgia, Moldova and Romania. The information includes CT scans, X-ray images, genome data, clinical and social data.

Moreover, for all patients in both subtasks we provided automatically extracted masks of the lungs to make further processing easier. These masks were extracted using the method described in [ 7 ]. Figure 1 shows one CT example with the provided mask as overlay.

Multi-drug Resistance Detection For this subtask, a dataset of 3D CT images was used along with a set of clinically relevant metadata. The dataset includes only HIV-negative patients with no relapses. Each patient was classi ed into one the two classes: drug sensitive (DS) or multi-drug resistant (MDR) cases. 5 http://tbportals.niaid.nih.gov/ A patient was considered DS if tuberculosis bacteria were sensitive to all the antituberculosis drugs tested. In case of MDR tuberculosis the corresponding drug susceptibility tests showed resistance to at least Isoniazid and Rifampicin { the two most powerful anti-TB drugs. In reality there are several levels of resistances but to make the task as easy as possible only these two most representative classes were retained. Tuberculosis Type Classi cation The dataset used in this subtask includes chest CT scans of TB patients along with the TB type. Figure 2 shows one example for each of the ve TB types. The participants were allowed to submit up to 10 runs to the tuberculosis task for each of the two sub tasks. In the case of the MDR tasks, participants had to provide, for each patient in the test set, the probability between 0 and 1 of belonging to the MDR class. These probabilities were used to build Receiver Operating Characteristic (ROC) curves. Since the MDR dataset was not perfectly balanced and had a relative small size, we decided to use the Area Under the ROC Curve (AUC) to rank the participant runs. Moreover, we provided the accuracy of the binary classi cation using a standard threshold of 0.50.

In the case of the TBT task, the participants had to predict the TB type of each patient, and submit a run containing a categorical label in the set f1, 2, 3, 4, 5g. For this task Cohen's Kappa was provided for each run along with the weighted average accuracy along the classes. The former measure is not sensitive to unbalanced datasets as in this task. 94 teams registered for the tuberculosis task and 48 signed the end user agreement. Finally, 9 teams from 9 countries submitted a run for at least one of the two subtasks. Table 3 shows the list of participants and the task(s) where they participated. 3

Results

This section provides the results obtained by the participants in each of the subtasks. 3.1

MDR Detection

It should be noticed that the task of image-based recognition of MDR for tuberculosis is very di cult and challenging. It was not clear from the start whether this task is actually possible with visual data alone. There are several studies dedicated to this problem that reported presence of statistically significant links between drug resistance status and various features of the patient lungs [8{11]. However, none of the studies reported any precise classi cation of drug resistance with accuracy far beyond the level of statistical signi cance.

Among the submitted runs, the three best results were achieved by the MedGIFT team. Methods based on a graph-model of the lungs were used for the description of lung CT appearance [ 12 ]. For di erent variations of the method the AUC calculated over the test set varied from 0.5112 to 0.5825. The SGEast team employed deep learning in both subtasks. Two-dimensional convolutional neural networks were used for describing the CT scans and recurrent neural networks were used for image classi cation [ 13 ]. Combined with preprocessing procedures such as image slicing and data augmentation the method resulted in 0.5620 AUC value for their best run. The UIIP team used an image description method based on the calculation of co-occurrence matrices of 3D supervoxels which allowed to achieve an AUC of 0.5415 [ 14 ]. The best performing result in terms of classi cation accuracy was achieved by the HHU DBS team by using deep convolutional networks which operate directly in 3D [ 15 ]. The method demonstrated a classi cation accuracy of 56.8% with an AUC of 0.5297 for the best run. The Batmanlab used a 3D intensity-based supervoxel image representation and various texture features in each supervoxel and achieved their maximum performance at 0.5241 AUC [16]. 2D deep learning techniques were used by MEDGIFT UPB along with CT image slicing and data augmentation, which resulted in an AUC of 0.5184 for their best run [17]. The Aegean Tubercoliosis group obtained an AUC of 0.4833 with their single run. However, no details on their technique were provided. Finally, the Bioinformatics UA team also used 3D convolutional neural networks in their study and achieved 46.0% of classi cation accuracy with an AUC of 0.4648 in the MDR subtask [18]. The authors believe that using data augmentation and weight regularization may potentially improve the results. 3.2

Tuberculosis Type Classi cation

Table 5 shows the results obtained for the tuberculosis type subtask. The runs were evaluated on the test set of images using the unweighted Cohen's Kappa coe cient and classi cation accuracy. The results are sorted by Cohen's Kappa 0.9 0.8 0.7 e ta0.6 r e v i its0.5 o p rue0.4 T 0.3 0.2 0.1 00 MedGIFT SGEast UIIP HHU DBS BatmanLab MEDGIFT UPB Aegean Tubercoliosis BioinformaticsUA 0.1 0.2 0.3 0.4 coe cient in descending order. In addition, Figure 4 contains the confusion matrices of each team's best run. These matrices show the percentage of patients from each class classi ed in each TB type. When analyzing these matrices, we found that the best performance per TB type is found in the rst 4 teams (SGEast, MEDGIFT UPB, Image Processing, and UIIP). These methods seem to be complementary, being expert each one in classifying a di erent TB type. Thus, the organizers of the task have performed a late fusion with the best run from these 4 best teams. The results of the corresponding fusion are included at the end of both Table 5 and Figure 4. In the tuberculosis type subtask, most of the teams used the same set of methods as they used for MDR detection subtask. The best results were achieved by the SGEast and MEDGIFT UPB groups with Cohen's Kappa of 0.2438 and 0.2329 respectively for their best runs. The Image Processing group used a technique based on splitting CT images into sets of small 2D patches followed by classi cation of image patches via convolutional neural networks [19]. The corresponding submitted run scored 0.2187 for Cohen's Kappa. The UIIP team used a di erent approach for TB type classi cation that uses extended multi-sort co-occurrence matrices of voxels in 3D images. The only submitted run resulted in a Kappa score of 0.1956. The best runs of MedGIFT, BatmanLab and BioinformaticsUA groups achieved Cohen's Kappa values of 0.1623, 0.1533, and 0.0222, respectively. Finally, the fusion of the best runs of the 4 best teams achieved a 0.3263 Cohen's Kappa value. This result is an important improvement with respect the best participant run and shows how complementary the approaches of these four participants are. However, when analyzing its confusion matrix in Figure 4, this fusion method mainly improved the performance for T1. In all the other classes, the performance is below the best performance achieved in the corresponding class by at least one of the fused methods. In any case, the average accuracy obtained it is also superior (0.4867) to all submitted runs by an important margin. 4

Discussion and Conclusions

The results obtained by the participants con rm the di culty of the MDR task. Independently of the technique applied, all runs remained relatively close to the performance of a random classi er, meaning that there is likely a high potential for improvements. One explanation is that it is not possible to detect MDR patients based on images alone, and maybe additional data on the patients need to be combined with visual information. The results above random encourage to nd and add more cases to this dataset and continue exploring the detection of drug resistance based on images, as a much larger training data set can maybe better cover the di erent subtypes, as can potentially the addition of multimodal information. On the other hand, there are other existing forms of TB drug resistance which are not considered with this study. For example, quick detection of Extensively drug-resistant (XDR) tuberculosis is also of a great scienti c and practical interest.

The TB subtask results are clearly based on the suitability of the imaging techniques for a speci c task. The best runs seem to exploit quite di erent aspects and each of them is best for one speci c class. This complementarity is highlighted by the very good results of a simple late fusion of the four best runs. However, there is still room for improvement, especially in the types Focal (T2), Tuberculoma (T3), Miliary (T4), and Fibro-cavernous (T5). For this particular task, deep learning methods worked better than other approaches, obtaining the 6 best results. Nonetheless, the late fusion performed by the organizers combining the prediction of three deep learning techniques and one based on 3D texture features surpassed the best participant run. This suggests that di erent TB types may need to be described by di erent approaches and such an approach can lead to optimal results.

2017 was the rst year of the ImageCLEF TB task. The number of registered participants and the results obtained show the interest of the community in this task, and encourages the inclusion of the TB task in the upcoming editions of ImageCLEF. There is still much to be learned and a detailed analysis of the results should help with this.

Acknowledgements

This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030{146804) and by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project OISE-16-62631-1. 16. Dicente Cid, Y., Muller, H., Batmanghelich, K.: BatmanLab in the ImageCLEF tuberculosis task 2017. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceur-ws.org> (September 11-14 2017) 17. Stefan, L.D., Dicente Cid, Y., Jimenez-Del-Toro, O., Ionescu, B., Muller, H.: Finding and classifying tuberculosis types for a targeted treatment: MedGIFT-UPB participation in ImageCLEFtuberculosis (2017). In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceurws.org> (September 11-14 2017) 18. Silva, J.a.F., Silva, J.M., Pinho, E., Costa, C.: 3D-CNN in drug resistance detection and tuberculosis classi cation. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceur-ws.org> (September 11-14 2017) 19. Gao, X., Qian, Y.: Application of deep learning neural network for classi cation of TB lung CT images based on patches. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org <http://ceur-ws.org> (September 11-14 2017)

References

1. Kalpathy-Cramer , J. , Garc a Seco de Herrera, A. , Demner-Fushman , D. , Antani , S. , Bedrick , S. , Muller, H.: Evaluating performance of biomedical image retrieval systems: Overview of the medical image retrieval task at ImageCLEF 2004{2014 . Computerized Medical Imaging and Graphics 39 ( 0 ) ( 2015 ) 55 { 61

2. Muller, H., Clough , P. , Deselaers , T. , Caputo , B., eds.: ImageCLEF { Experimental Evaluation in Visual Information Retrieval . Volume 32 of The Springer International Series On Information Retrieval. Springer, Berlin Heidelberg ( 2010 )

3. Ionescu , B. , Muller, H., Villegas , M. , Arenas , H. , Boato , G. , Dang-Nguyen , D.T. , Dicente Cid , Y. , Eickho , C. , Garcia Seco de Herrera , A. , Gurrin , C. , Islam , B. , Kovalev , V. , Liauchuk , V. , Mothe , J. , Piras , L. , Riegler , M. , Schwall , I. : Overview of ImageCLEF 2017: Information extraction from images . In: Experimental IR Meets Multilinguality, Multimodality, and Interaction 8th International Conference of the CLEF Association , CLEF 2017 . Volume 10456 of Lecture Notes in Computer Science., Dublin, Ireland, Springer (September 11 -14 2017 )

4. Villegas , M. , Muller, H., Garcia Seco de Herrera, A. , Schaer , R. , Bromuri , S. , Gilbert , A. , Piras , L. , Wang , J. , Yan , F. , Ramisa , A. , Dellandrea , A. , Gaizauskas , R. , Mikolajczyk , K. , Puigcerver , J. , Toselli , A.H. , Sanchez , J.A. , Vidal , E.: General overview of ImageCLEF at the CLEF 2016 labs . In: CLEF 2016 Proceedings. Lecture Notes in Computer Science , Evora. Portugal, Springer ( September 2016 )

5. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Garc a Seco de Herrera, A. , Bromuri , S. , Amin , M.A. , Kazi Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , Roldan Garc a , M.d.M.: General overview of ImageCLEF at the CLEF 2015 labs . In: Working Notes of CLEF 2015. Lecture Notes in Computer Science . Springer International Publishing ( 2015 )

6. Caputo , B. , Muller, H., Thomee , B. , Villegas , M. , Paredes , R. , Zellhofer , D. , Goeau , H. , Joly , A. , Bonnet , P. , Martinez Gomez , J. , Garcia Varea , I. , Cazorla , C. : ImageCLEF 2013: the vision, the data and the open challenges . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

7. Dicente Cid , Y. , Jimenez-del- Toro , O. , Depeursinge , A. , Muller, H.: E cient and fully automatic segmentation of the lungs in CT volumes . In Orcun Goksel, Jimenez-del- Toro , O. , Foncubierta-Rodriguez , A. , Muller, H., eds. : Proceedings of the VISCERAL Challenge at ISBI. Number 1390 in CEUR Workshop Proceedings (Apr 2015 )

8. Cha , J. , Lee , H.Y. , Lee , K.S. , Koh , W.J. , Kwon , O.J. , Yi , C.A. , Kim , T.S. , Chung , M.J.: Radiological ndings of extensively drug-resistant pulmonary tuberculosis in non-AIDS adults: comparisons with ndings of multidrug-resistant and drugsensitive tuberculosis . Korean journal of radiology 10(3) ( 2009 ) 207 { 216

9. Lee , E.S. , Park , C.M. , Goo , J.M. , Yim , J.J. , Kim , H.R. , Lee , H.J. , Lee , I.S. , Im , J.G. : Computed tomography features of extensively drug-resistant pulmonary tuberculosis in non-HIV-infected patients . Journal of computer assisted tomography 34(4) ( 2010 ) 559 { 563

10. Kovalev , V. , Liauchuk , V. , Safonau , I. , Astrauko , A. , Skrahina , A. , Tarasau , A. : Is there any correlation between the drug resistance and structural features of radiological images of lung tuberculosis patients ? In: Computer Assisted Radiology - 27th International Congress and Exhibition (CARS-2013) . Volume 8 ., Springer, Heidelberg ( 2013 ) 18 { 20

11. Kovalev , V. , Liauchuk , V. , Kalinouski , A. , Rosenthal , A. , Gabrielian , A. , Skrahina , A. , Astrauko , A. , Tarasau: Utilizing radiological images for predicting drug resistance of lung tuberculosis . In: Computer Assisted Radiology - 27th International Congress and Exhibition (CARS-2015) . Volume 10 ., Springer, Barcelona ( 2015 ) 129 { 130

12. Dicente Cid , Y. , Batmanghelich , K. , Muller, H.: Textured graph-model of the lungs for tuberculosis type classi cation and drug resistance prediction: participation in ImageCLEF 2017 . In: CLEF2017 Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

13. Sun , J. , Chong , P. , Tan , Y.X.M. , Binder , A. : ImageCLEF 2017: ImageCLEF tuberculosis task - the SGEast submission . In: CLEF2017 Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

14. Liauchuk , V. , Kovalev , V.: ImageCLEF 2017: Supervoxels and co-occurrence for tuberculosis CT image classi cation . In: CLEF2017 Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

15. Braun , D. , Singhof , M. , Tatusch , M. , Conrad , S. : Convolutional neural networks for multidrug-resistant and drug-sensitive tuberculosis dinstinction . In: CLEF2017 Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )