Overview of ImageCLEFtuberculosis 2018 { Detecting Multi-Drug Resistance, Classifying Tuberculosis Types and Assessing Severity Scores

Abstract

ImageCLEF is the image retrieval task of the Conference and Labs of the Evaluation Forum (CLEF). ImageCLEF has historically focused on the multimodal and language-independent retrieval of images. Many tasks are related to image classi cation and the annotation of image data as well as the retrieval of images. The tuberculosis task was held for the rst time in 2017 and had a very encouraging participation with 9 groups submitting results to these very challenging tasks. In 2018 there was a slightly higher participation. Three tasks were proposed in 2018: (1) the detection of drug resistances among tuberculosis cases, (2) the classi cation of the cases into ve types of tuberculosis and (3) the assessment of a tuberculosis severity score. Many di erent techniques were used by the participants ranging from Deep Learning to graph-based approaches and best results were obtained by a variety of approaches with no clear technique dominating. Both, the detection of drug resistances and the classi cation of tuberculosis types had similar results than in the previous edition, the former remaining as a very difcult task. In the case of the severity score task, the results support the suitability of assessing the severity based only on the CT image, as the results obtained were very good.

Keywords

• Tuberculosis
• Computed Tomography
• Image Classi cation
• Drug Resistance
• Severity Scoring
• 3D Data Analysis

-

ImageCLEF4 is the image retrieval task of CLEF (Conference and Labs of the Evaluation Forum). ImageCLEF was rst held in 2003 and in 2004 a medical task was added that has been held every year since then [1{4]. More information

4 http://www.imageclef.org/

on the other tasks organized in 2018 can be found in [ 5 ] and the past editions are described in [6{9].

Tuberculosis (TB) is a bacterial infection caused by a germ called Mycobacterium tuberculosis. About 130 years after its discovery, the disease remains a persistent threat and a leading cause of death worldwide [ 10 ]. This bacteria usually attacks the lungs, but it can also damage other parts of the body. Generally, TB can be cured with antibiotics. However, the greatest disaster that can happen to a patient with TB is that the organisms become resistant to two or more of the standard drugs. In contrast to drug sensitive (DS) TB, its multi-drug resistant (MDR) form is much more di cult and expensive to recover from. Thus, early detection of the MDR status is fundamental for an e ective treatment. The most commonly used methods for MDR detection are either expensive or take too much time (up to several months) to really help in this scenario. Therefore, there is a need for quick and at the same time cheap methods of MDR detection. In 2017, ImageCLEF organized the rst challenge based on Computed Tomography (CT) image analysis of TB patients [ 11 ], with a dedicated subtask for the detection of MDR cases. The classi cation of TB subtypes was also proposed in 2017. This is another important task for TB analysis since di erent types of TB should be treated in di erent ways. Both subtasks were also proposed in the 2018 edition where we extended their respective datasets. Moreover, a new subtask was added based on assessing a severity score of the disease given a CT image.

This article rst describes the three tasks proposed around TB in 2018. Then, the datasets, evaluation methodology and participation are detailed. The results section describes the submitted runs and the results obtained for the three subtasks. A discussion and conclusion section ends the paper. 2 2.1

Tasks, Datasets, Evaluation, Participation

The Tasks in 2018

Three subtasks were organized in 2018. Two were common with the 2017 edition and one new subtask was added: { Multi-Drug Resistance detection (MDR subtask); { Tuberculosis Type classi cation (TBT subtask); { Severity Scoring assessment (SVR subtask).

This section gives an overview of each of the three subtasks.

Multi-drug Resistance Detection: As in 2017, the goal of the MDR subtask was to assess the probability of a TB patient having a resistant form of TB based on the analysis of a chest CT scan alone. The dataset for this subtask was increased from the previous year but the subtask remained as a binary classi cation problem even though several levels of resistances exist. Tuberculosis Type Classi cation: This subtask is also common with the 2017 edition and, like in the MDR subtask, we increased the dataset. The goal of the TBT subtask is to automatically categorize each TB case into one of the following ve TB types: In ltrative, Focal, Tuberculoma, Miliary, and Fibrocavernous. The distribution of cases among the classes is not balanced but the distributions are similar in the training and the test data.

Severity Scoring: This subtask aims at assessing a TB severity score based only on a chest CT image. The severity score is a cumulative score of severity of a TB case assigned by a medical doctor. Originally, the score varied from 1 ("critical/very bad") to 5 ("very good"). In the process of scoring, the medical doctors considered many factors like pattern of the lesions, results of microbiological tests, duration of treatment, patient age and other criteria. 2.2

Datasets

For each of the three subtasks, a separate dataset was provided, all containing 3D CT images stored in the NIfTI (Neuroimaging Informatics Technology Initiative) le format with slice resolution of 512 512 pixels and a number of slices varying from about 50 to 400. A set of relevant meta-data such as age and gender was provided for each subtask. The entire dataset including CT images and associated meta-data were provided by the Republican Research and Practical Center for Pulmonology and Tuberculosis that is located in Minsk, Belarus. The data were collected in the framework of several projects that aim at the creation of information resources on lung TB and drug resistance challenges. The projects were conducted by a multi-disciplinary team and funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH), U.S. Department of Health and Human Services, USA, through the Civilian Research and Development Foundation (CRDF). The dedicated web-portal5 developed in the framework of the projects stores information of more than 940 TB patients from ve countries: Azerbaijan, Belarus, Georgia, Moldova and Romania. The information includes CT scans, X-ray images, genome data, clinical and social data.

In the framework of the ImageCLEF 2018 TB task, automatically extracted masks of the lungs were provided for all CT images. These masks were extracted using the method described in [ 12 ]. The segmentations were analyzed based on the number of lungs found and the size ratio of the lungs in a supervised manner. Only those segmentations with anomalies on these two metrics were visualized and evaluated accordingly. A total of 32 images out of 2,287 presented a problematic mask, 8 including areas outside the lungs and 24 containing only one lung. The 8 inaccurate masks were corrected by fusing the above mentioned method and the registration-based segmentation used in [ 13 ]. The other 24 masks (20 from the TBT subtask and 4 from the MDR subtask) could not be properly

5 http://tbportals.niaid.nih.gov/

labeled due to the size and/or damage of one lung. In these cases, the masks provided to the participants only contained one label (right lung). Multi-drug Resistance Detection The dataset for this task is an extension of the one used in the 2017 edition. Particularly, the training and test sets of this subtask were extended by adding patients with extensively drug-resistant (XDR) TB, which is a rare and more severe subtype of MDR TB. Along with the 3D CT images and lung masks, the age and gender of each patient were provided. The dataset includes only HIV-negative patients with no relapses. Each patient was classi ed into one the two classes: drug sensitive (DS) or multi-drug resistant (MDR). A patient was considered DS if the TB bacteria was sensitive to all the anti-tuberculosis drugs tested. All XDR patients were considered to belong to the MDR class. Table 1 contains the number of patients in each set. Tuberculosis Type Classi cation The dataset used in this subtask includes chest CT scans of TB patients along with the TB type and patient age at the moment of the scan. Like the MDR dataset, the TBT 2017 dataset was extended for the 2018 edition. In this case, new CT scans of the same patients involved in 2017 were added and also some CT images of new patients. In the TBT 2018 dataset, for each patient there are between 1 and 9 CT scans acquired at di erent time points. All scans of the same patient were diagnosed with the same TB type by expert radiologists. Figure 1 shows one example for each of the ve TB types. Moreover, Figure 2 shows examples of two patients with three CT scans each. The CT slices in both gures are shown using a Houns eld Unit (HU) window with center at -500 HU and width of 1400 HU. The number of CT scans and patients in each TB type set are shown in Table 2.

Severity Scoring The data for the SVR subtask includes 279 CT scans with known TB severity scores ranging from 1 to 5 assigned by medical doctors. Each CT scan corresponds to a speci c TB patient. To treat this subtask as a binary classi cation problem, the severity scores were grouped so that values 1, 2 and 3 corresponded to "high severity" class, and values 4 and 5 corresponded to "low severity". Table 3 contains the number of patients of each severity class in the sets.

In ltrative

Focal

Tuberculoma Miliary

Fibro{cavernous Similar to 2017, the participants were allowed to submit up to 10 runs to each of the three TB subtasks. In the case of the MDR subtask, the participants had to 46 years old 53 years old 54 years old 55 years old 58 years old 59 years old provide the probability for the TB cases to belong to the MDR class ranging from 0 to 1. These probabilities were used to build Receiver Operating Characteristic (ROC) curves. Since the MDR dataset was not perfectly balanced and had a relatively small size, Area Under the ROC Curve (AUC) was used to evaluate the participant runs. We provided the accuracy of the binary classi cation using a standard threshold of 0.50.

In the case of the TBT task, the participants had to predict the TB type of each patient, and submit a run containing a category label in the set f1, 2, 3, 4, 5g. Considering that a high number of patients in the dataset had multiple CT scans with the same TB type, the evaluation was performed patient-wise. Cohen's Kappa coe cient was provided for each run along with the 5-class prediction accuracy. Cohen's Kappa is not sensitive to unbalanced datasets, which is the case for the data used here.

The runs submitted for the severity scoring subtask were evaluated in two ways. One used the original severity scores from 1 to 5 and the task was to predict those numerical scores as precise as possible (a regression problem). Here, Root Mean Square Error (RMSE) was computed between the ground truth severity and the predicted scores provided by the participants. Alternatively, the original severity score was transformed into two classes, where scores from 1 to 3 corresponded to "high severity" and the 4 and 5 scores corresponded to the "low severity" class. In this case the participants had to provide the probability of TB cases to belong to the "high severity" class. The corresponding results were evaluated using AUC. 2.4

Participation

In 2018 there were 85 registered teams and 33 signed the end user agreement. Finally, 11 groups from 9 countries participated in one or more subtasks and submitted results. These numbers are similar to 2017, where there were 94 registered teams, 48 that signed the end user agreement, and 9 teams from 9 countries submitting results. Table 4 shows the list of participants and the subtasks where they participated. One of the groups (HHU-DBS) participated in two subtasks with di erent approaches developed by a di erent set of authors. Therefore, their approaches are referred as HHU-DBS 1 and HHU-DBS 2 in the following sections. ( ) The HHU-DBS group participated with di erent approaches in the MDR and SVR subtasks. Therefore, the group name is split into HHU-DBS 1 and HHU-DBS 2 respectively in the following sections. 3

Results

This section provides the results obtained by the participants in each of the subtasks. 3.1 0.8

SDVAHCS/UCSD

UniversityAlicante

ed UIIP_BioM

It is worth to notice that the image-based detection task of MDR TB remains very challenging and so far has no solution with a su ciently high prediction accuracy for being useful in clinical practice. Recent articles report the presence of statistically signi cant links between drug resistance and multiple thick-walled caverns [ 14 ]. However, computerized methods show a performance of image-based MDR TB detection barely beyond the level of statistical signi cance compared to a random classi er [ 6, 15, 16 ].

The best result in terms of AUC was achieved by VISTA@UEvora team with an AUC of 0.6178 [ 17 ]. The team used conventional approaches for the extraction of quantitative image descriptors, such as statistical moments, fractal dimension, gray-level co-occurrence matrices and their derivative features. A set of conventional classi cation methods was used for prediction in all the three subtasks. Their best run in terms of classi cation accuracy (0.5763) ranked 3rd place among the participant runs and is not the same run that had the best AUC. The second highest AUC of 0.6114 was achieved by the San Diego VA HCS/UCSD [ 18 ] with an approach based on splitting the 3D CT scans into a set of 2D images and using a pre-trained ResNeXt deep network for classi cation. This run achieved the highest MDR detection accuracy (0.6144). The third highest AUC was obtained by HHU-DBS 1 [ 19 ]. They used 3D deep Convolutional Neural Networks (CNNs) combined with decision trees and obtained 0.5810 AUC and 0.5720 classi cation accuracy with their best run. The UniversityAlicante group used two approaches: one based on 2D CNNs and the other based on Optical Flow (OF) [ 20 ]. The best AUC among this group's runs was obtained using only patient age and gender information and ranked 10th among all participant runs with a AUC of 0.5669. Other runs obtained lower AUC. This OF-based approach for CT image analysis resulted in an accuracy of 0.5339 and ranked 20th. The single run submitted by the UIIP BioMed group ranked 14th in AUC and 36th in accuracy with an AUC of 0.5558 and an accuracy of 0.4576 [ 21 ]. A technique for automatic detection of lesions of di erent types in a six-region division of the CT lung volume was used. A separate dataset with labeled lesions in CT was used for training the lesion detection algorithm. A Random Forest (RF) classi er was used for the prediction of the nal classes and scores in all three subtasks. Methods based on a graph-model of the lungs and 3D texture analysis were used by MedGIFT group [ 22 ]. Their best runs resulted in the 22nd highest AUC (0.5237) and the 2nd highest accuracy (0.5932). Finally, the LIST group used a hybrid approach that combined 3D CNNs with linear SVM classi ers for MDR detection and TB type classi cation [ 23 ]. The single run submitted by the group obtained an AUC of 0.5029 and an accuracy of 0.4576 and ranked 28th and 37th, respectively. The information about age and gender of TB patients was used only by two participating groups: HHU-DBS 1 and UIIP BioMed. 3.2

Tuberculosis Type Classi cation

Table 6 shows the results obtained for the TBT subtask. The runs were evaluated on the test set of images using the unweighted Cohen Kappa coe cient and overall classi cation accuracy. The results are sorted by Cohen's Kappa in descending order. Figure 4 shows the highest Kappa values achieved by the participants. The true positive rates of the di erent TB types are shown in Figure 5.

In the TBT subtask, most of the teams used the same methods as they used for the MDR detection. The best result in terms of both Kappa and classi cation accuracy was achieved by the UIIP BioMed group with the use of a lesionbased TB descriptor and a RF classi er. The run resulted in a Kappa of 0.2312 and a classi cation accuracy of 0.4227. Instead of using all the available CT series, this group only used the rst scan of a patient for the classi cation of the TB type. The second highest Kappa was achieved by the fau ml4cv group that participated only in the TBT subtask [ 24 ]. An ensemble of 3D CNNs was used, achieving a Kappa of 0.1736 and an accuracy of 0.3533 with their best run. The graph-based approach of the MedGIFT team resulted in the 2nd best classi cation accuracy (0.3849) and the 3rd highest Kappa (0.1706). The best runs of VISTA@UEvora, San Diego VA HCS/UCSD, UniversityAlicante and LIST resulted in Kappa values of 0.1664, 0.1474, 0.0204, and -0.0024 respectively. The MostaganemFSEI group participated in the TBT classi cation and the SVR subtasks. The algorithm employed by them was based on splitting the 3D CT scans into 2D slices, extracting semantic descriptors using a trained CNN and applying conventional classi cation methods [ 25 ]. They obtained a Kappa of 0.0629 and an accuracy of 0.2744. It is worth to highlight that only the fau ml4cv and San Diego VA HCS/UCSD groups obtained a true positive rate higher than a random classi er in all ve TB types (see Figure 5). 3.3 The results obtained for the severity scoring subtask are shown in Table 7. The best RMSE achieved by the participating groups and the corresponding AUCs are shown in Figures 6 and 7. The best results in terms of regression were obtained by the UIIP BioMed group with an RMSE of 0.7840, which also achieved the 6th best classi cation result with an AUC of 0.7025. The highest classi cation result was achieved by the MedGIFT group with an AUC of 0.7708. The MedGIFT group's best regression obtained an RMSE of 0.8513, which is the second best result. The third best RMSE (0.8883) was obtained by the VISTA@UEvora group. The same run ranked on the 21st place for classi cation a pp 0 a K -0.5

SDVAHCS/UCSD with an AUC of 0.6239. The third best result for classi cation was obtained by the San Diego VA HCS/UCSD group with an AUC of 0.6984, which corresponds to the 7th best result. Their best regression is an RMSE of 1.2153, which is at rank 30. The HHU-DBS 2 team used a feature-based approach for scoring the severity of TB based on a set of conventional methods [26]. The approach employed image binarization and extraction of features including the presence of calci cations, lung wateriness, cavities, infection ratio, HU histograms and lung shape to characterize the volumes. The group obtained the 10th best RMSE (0.9626) and 8th best AUC (0.6862). The MostaganemFSEI group achieved an RMSE of 0.9721 and an AUC of 0.6127. Middlesex University participated only in the SVR subtask. The group employed an approach based on using deep residual learning, training on a set of overlapping 128 128 depth blocks, assessing the TB severity for each block and gathering the results [27]. This allowed to achieve an RMSE of 1.0921 and an AUC of 0.6534 that correspond to the 24th and 14th positions. It is important to highlight that all groups obtained an AUC higher than a random classi er (AUC of 0.50) with all their runs. 1.5 E S RM 1 0.5 0 1 0.8 Similar to 2017, the results obtained by the participants in the MDR detection subtask demonstrate that the task of a fully automatic image-based detection of drug resistance is extremely di cult. Despite the addition of XDR TB cases into the dataset and the inclusion of information about patient age and gender, the MDR detection performance still remains at a level relatively close to a random classi cation with the highest reached AUC of 0.6178 and a 61.4% prediction accuracy. The overall increase of prediction performance with respect to the 2017 edition might be caused by the addition of more severe cases with XDR TB into the dataset. Using information about patient age and gender could also improve the MDR detection results as suggested by the baseline submitted by UniversityAlicante group [ 20 ].

In the second subtask, the overall results of TB type classi cation are slightly worse than in 2017. This might be caused by the decreased balance of TB classes in the dataset. Using more than one CT scan per patient could also confuse prediction methods and worsen the nal results. However there is a certain improvement in prediction of class T2 (Focal TB) demonstrated by most of the participants' results.

The results of SVR subtask are encouraging, since the actual assessment of the TB severity score is done using various clinical information sources, not only CT image data. Most of the results achieved by the participants obtained a RMSE of the severity score below 1 in a 5-grade scoring system. The best results obtained using only CT volumes are close to the results reported in [28], where the authors used clinical and laboratory data including drug resistance, presence of TB symptoms, etc. in addition to the images. Extension of the dataset and usage of clinical and laboratory data is expected to improve the severity scoring results.

Overall, the 2018 edition of the ImageCLEF TB task showed an improvement with respect to the 2017 edition in terms of number of participants, data provided, results obtained and the variety of methods proposed. This shows a high interest in this topic and also the importance of the data that were generated.

Acknowledgements

This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030{146804) and by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-17-63599-1 "Year 6: Belarus TB Database and TB Portals". 26. Bogomasov, K., Himmelspach, L., Klassen, G., Tatusch, M., Conrad, S.: Featurebased approach for severity scoring of lung tuberculosis from CT images. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, CEUR-WS.org <http://ceur-ws.org> (September 10-14 2018) 27. Gao, X., James-Reynolds, C., Currie, E.: Scoring TB severity with an enhanced deep residual learning depth-resnet. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, CEUR-WS.org <http://ceur-ws.org> (September 10-14 2018) 28. Kovalev, V., Liauchuk, V., Skrahina, A., Astrauko, A., Rosenthal, A., Gabrielian, A.: Examining the utility of clinical, laboratory and radiological data for scoring severity of pulmonary tuberculosis. In: Computer Assisted Radiology and Surgury - 32nd International Congress and Exhibition (CARS-2018). Volume 13., Springer, Heidelberg (2018) 143{144

References

1. Kalpathy-Cramer , J. , Garc a Seco de Herrera, A. , Demner-Fushman , D. , Antani , S. , Bedrick , S. , Muller, H.: Evaluating performance of biomedical image retrieval systems: Overview of the medical image retrieval task at ImageCLEF 2004{2014 . Computerized Medical Imaging and Graphics 39 ( 0 ) ( 2015 ) 55 { 61

2. Muller, H., Clough , P. , Deselaers , T. , Caputo , B., eds.: ImageCLEF { Experimental Evaluation in Visual Information Retrieval . Volume 32 of The Springer International Series On Information Retrieval. Springer, Berlin Heidelberg ( 2010 )

3. Garc a Seco de Herrera, A. , Schaer , R. , Bromuri , S. , Muller, H.: Overview of the ImageCLEF 2016 medical task . In: Working Notes of CLEF 2016 ( Cross Language Evaluation Forum) . (September 2016 )

4. Muller, H., Clough , P. , Hersh , W. , Geissbuhler , A. : ImageCLEF 2004 {2005: Results experiences and new ideas for image retrieval evaluation . In: International Conference on Content{Based Multimedia Indexing (CBMI 2005 ), Riga, Latvia, IEEE ( June 2005 )

5. Ionescu , B. , Muller, H., Villegas , M., de Herrera , A.G.S. , Eickho , C. , Andrearczyk , V. , Cid , Y.D. , Liauchuk , V. , Kovalev , V. , Hasan , S.A. , Ling , Y. , Farri , O. , Liu , J. , Lungren , M. , Dang-Nguyen , D.T. , Piras , L. , Riegler , M. , Zhou , L. , Lux , M. , Gurrin , C. : Overview of ImageCLEF 2018: Challenges, datasets and evaluation. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018 ), Avignon, France, LNCS Lecture Notes in Computer Science , Springer (September 10 -14 2018 )

6. Ionescu , B. , Muller, H., Villegas , M. , Arenas , H. , Boato , G. , Dang-Nguyen , D.T. , Dicente Cid , Y. , Eickho , C. , Garcia Seco de Herrera , A. , Gurrin , C. , Islam , B. , Kovalev , V. , Liauchuk , V. , Mothe , J. , Piras , L. , Riegler , M. , Schwall , I. : Overview of ImageCLEF 2017: Information extraction from images . In: Experimental IR Meets Multilinguality, Multimodality, and Interaction 8th International Conference of the CLEF Association , CLEF 2017 . Volume 10456 of Lecture Notes in Computer Science., Dublin, Ireland, Springer (September 11 -14 2017 )

7. Villegas , M. , Muller, H., Garcia Seco de Herrera, A. , Schaer , R. , Bromuri , S. , Gilbert , A. , Piras , L. , Wang , J. , Yan , F. , Ramisa , A. , Dellandrea , A. , Gaizauskas , R. , Mikolajczyk , K. , Puigcerver , J. , Toselli , A.H. , Sanchez , J.A. , Vidal , E.: General overview of ImageCLEF at the CLEF 2016 labs . In: CLEF 2016 Proceedings. Lecture Notes in Computer Science , Evora. Portugal, Springer ( September 2016 )

8. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Garc a Seco de Herrera, A. , Bromuri , S. , Amin , M.A. , Kazi Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , Roldan Garc a , M.d.M.: General overview of ImageCLEF at the CLEF 2015 labs . In: Working Notes of CLEF 2015. Lecture Notes in Computer Science . Springer International Publishing ( 2015 )

9. Caputo , B. , Muller, H., Thomee , B. , Villegas , M. , Paredes , R. , Zellhofer , D. , Goeau , H. , Joly , A. , Bonnet , P. , Martinez Gomez , J. , Garcia Varea , I. , Cazorla , C. : ImageCLEF 2013: the vision, the data and the open challenges . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

10. World Health Organization, et al.: Global tuberculosis report 2016 . ( 2016 )

11. Dicente Cid , Y. , Kalinovsky , A. , Liauchuk , V. , Kovalev , V. , , Muller, H.: Overview of ImageCLEFtuberculosis 2017 - predicting tuberculosis type and drug resistances . In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

12. Dicente Cid , Y. , Jimenez-del- Toro , O. , Depeursinge , A. , Muller, H.: E cient and fully automatic segmentation of the lungs in CT volumes . In Orcun Goksel, Jimenez-del- Toro , O. , Foncubierta-Rodriguez , A. , Muller, H., eds. : Proceedings of the VISCERAL Challenge at ISBI. Number 1390 in CEUR Workshop Proceedings (Apr 2015 ) 31 { 35

13. Liauchuk , V. , Kovalev , V.: ImageCLEF 2017: Supervoxels and co-occurrence for tuberculosis CT image classi cation . In: CLEF2017 Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

14. Wang , Y.X.J. , Chung , M.J. , Skrahin , A. , Rosenthal , A. , Gabrielian , A. , Tartakovsky , M. : Radiological signs associated with pulmonary multi-drug resistant tuberculosis: an analysis of published evidences . Quantitative Imaging in Medicine and Surgery 8 ( 2 ) ( 2018 ) 161 { 173

15. Kovalev , V. , Liauchuk , V. , Safonau , I. , Astrauko , A. , Skrahina , A. , Tarasau , A. : Is there any correlation between the drug resistance and structural features of radiological images of lung tuberculosis patients ? In: Computer Assisted Radiology - 27th International Congress and Exhibition (CARS-2013) . Volume 8 ., Springer, Heidelberg ( 2013 ) 18 { 20

16. Kovalev , V. , Liauchuk , V. , Kalinouski , A. , Rosenthal , A. , Gabrielian , A. , Skrahina , A. , Astrauko , A. , Tarasau: Utilizing radiological images for predicting drug resistance of lung tuberculosis . In: Computer Assisted Radiology - 27th International Congress and Exhibition (CARS-2015) . Volume 10 ., Springer, Barcelona ( 2015 ) 129 { 130

17. Ahmed , M.S. , Obaidullah , S.M. , Jayatilake , M. , Goncalves , T. , Rato , L. : Texture analysis from 3D model and individual slice extraction for tuberculosis MDR detection, type classi cation and severity scoring . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

18. Gentili , A. : ImageCLEF2018: Transfer learning for deep learning with CNN for tuberculosis classi cation . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10- 14 2018 )

19. Tatusch , M. , Conrad , S. : Detection of multidrug-resistant tuberculosis using convolutional neural networks and decision trees . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

20. Llopis , F. , Fuster-Guillo , A. , Rico-Juan , J.R. , Azor n-Lopez, J. , Llopis , I. : Tuberculosis detection using optical ow and the activity description vector . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

21. Liauchuk , V. , Tarasau , A. , Snezhko , E. , Kovalev , V. , Gabrielian , A. , Rosenthal , A. : ImageCLEF 2018: Lesion-based TB-descriptor for CT image analysis . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

22. Dicente Cid , Y. , Muller, H.: Texture-based graph model of the lungs for drug resistance detection, tuberculosis type classi cation, and severity scoring: Participation in ImageCLEF 2018 tuberculosis task . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

23. Allaouzi , I. , Benamrou , B. , Ben Ahmed , M.: 3D-CNN in drug resistance detection and tuberculosis classi cation . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

24. Ishay , A. , Marques , O. : Ensemble of 3D CNNs with multiple inputs for tuberculosis type classi cation . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )

25. Hamadi , A. , Yagoub , D.E.: ImageCLEF 2018 : Semantic descriptors for tuberculosis CT image classi cation . In: CLEF2018 Working Notes. CEUR Workshop Proceedings , Avignon, France, CEUR-WS.org <http://ceur-ws. org> (September 10-14 2018 )