Overview of ImageCLEFtuberculosis 2019 | Automatic CT{based Report Generation and Tuberculosis Severity Assessment

Abstract

ImageCLEF is the image retrieval task of the Conference and Labs of the Evaluation Forum (CLEF). ImageCLEF has historically focused on the multimodal and language-independent retrieval of images. Many tasks are related to image classi cation and the annotation of image data as well as the retrieval of images. Since 2017, when the tuberculosis task started in ImageCLEF, the number of participants has kept growing. In 2019, 13 groups from 11 countries participated in at least one of the two subtasks proposed: (1) SVR subtask: the assessment of a tuberculosis severity score and (2) CTR subtask: the automatic generation of a CT report based on six relevant CT ndings. In this second edition of the SVR subtask the results support the assessment of a severity score based on the CT scan with up to 0.79 area under the curve (AUC) and 74% accuracy, so very good results. In addition, in the rst edition of the CTR subtask, impressive results were obtained with 0.80 average AUC and 0.69 minimum AUC for the six CT ndings proposed.

Keywords

• Tuberculosis
• Computed Tomography
• Image Classi cation
• Severity Scoring
• Automatic Reporting
• 3D Data Analysis

-

ImageCLEF5 is the image retrieval task of CLEF (Conference and Labs of the Evaluation Forum). ImageCLEF was rst held in 2003 and in 2004 a medical task was added that has been held every year since then [1{4]. More information on the other tasks organized in 2019 can be found in [ 5 ] and the past editions are described in [6{10].

Tuberculosis (TB) is a bacterial infection caused by a germ called Mycobacterium tuberculosis. About 130 years after its discovery, the disease remains a persistent threat and a leading cause of death worldwide [ 11 ]. This bacterium usually attacks the lungs but it can also damage other parts of the body. Generally, TB can be cured with antibiotics. However, the greatest problem that can happen to a patient with TB is that the organisms become resistant to two or more of the standard drugs. In contrast to drug sensitive (DS) TB, its multi-drug resistant (MDR) form is much more di cult and expensive to recover from. Thus, early detection of the MDR status is fundamental for an e ective treatment. The most commonly used methods for MDR detection are either expensive or take too much time (up to several months) to really help in this scenario. Therefore, there is a need for quick and at the same time cheap methods of MDR detection. In 2017, ImageCLEF organized the rst challenge based on Computed Tomography (CT) image analysis of TB patients [ 12 ], with a dedicated subtask for the detection of MDR cases. The classi cation of TB subtypes was also proposed in 2017. This is another important task for TB analysis since di erent types of TB should be treated in di erent ways. Both subtasks were also proposed in the 2018 edition where we extended their respective data sets. Moreover, a new subtask was added based on assessing a severity score of the disease given a CT image.

This article rst describes the two subtasks proposed around TB in 2019. Then, the data sets, evaluation methodology and participation are detailed. The results section describes the submitted runs and the results obtained for the two subtasks. A discussion and conclusion section ends the paper. 2

Tasks, Data Sets, Evaluation, Participation

2.1

The Tasks in 2019

Two subtasks were organized in 2019, one was common with the 2018 edition and one new subtask was added: { Severity score assessment (SVR subtask).

{ Automatic CT report generation (CTR subtask).

This section gives an overview of each of the two subtasks.

SVR - Severity Scoring: As in 2018, the goal of this subtask is to assess the severity based on the CT image and additional clinically relevant meta-data. The severity score is a cumulative score of severity of a TB case assigned by a medical doctor. Originally, the score varied from 1 ("critical/very bad") to 5 ("very good"). The original severity score was included as training meta-data but the nal score that participants had to assess was reduced to a binary category: "LOW" (scores 4 and 5) and "HIGH" (scores 1, 2 and 3).

CTR - CT Report: In this subtask the participants had to generate an automatic report based on the CT image. This report had to include the following CT ndings in binary form (0 or 1): Left lung a ected, right lung a ected, lung capacity decrease, presence of calci cations, presence of pleurisy and presence of caverns. 2.2

Data Sets

In 2019, both subtasks (SVR and CTR) used the same data set containing 335 chest CT scans of TB patients along with a set of clinically relevant meta-data, divided into 218 patients for training and 117 for testing. The selected metadata include the following binary measures: disability, relapse, symptoms of TB, comorbidity, bacillary, drug resistance, higher education, ex-prisoner, alcoholic, smoking, and severity. Table 1 details the distribution of patients within each label for the SVR and CTR subtasks.

Left Right Lung

High lung lung capacity Pres. of Pres. of Pres. of severity a ected a ected decrease calcif. pleurisy caverns Train 107 (49%) 156 (72%) 177 (81%) 64 (29%) 28 (13%) 16 (7%) 89 (41%) Test 55 (47%) 74 (63%) 95 (81%) 8 (7%) 60 (51%) 10 (9%) 40 (34%)

For all patients we provided 3D CT images with a slice size of 512 512 pixels and number of slices varying from about 50 to 400. All the CT images were stored in NIFTI le format with .nii.gz le extension (g-zipped .nii les). This le format stores raw voxel intensities in Houns eld units (HU) as well the corresponding image meta-data such as image dimensions, voxel size in physical units, slice thickness, etc. The entire dataset including the CT images and the associated meta-data were provided by the Republican Research and Practical Center for Pulmonology and Tuberculosis that is located in Minsk, Belarus. The data were collected in the framework of several projects that aim at the creation of information resources on the lung TB and drug resistance challenges. The projects were conducted by a multi-disciplinary team and funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH), U.S. Department of Health and Human Services, USA, through the Civilian Research and Development Foundation (CRDF). The dedicated web-portal6 developed in the framework of the projects stores information of more than 940 TB patients from ve countries: Azerbaijan, Belarus, Georgia, Moldova and Romania. The information includes CT scans, X-ray images, genome data, clinical and social data. 6 http://tbportals.niaid.nih.gov/

For all patients we provided automatically extracted masks of the lungs obtained using the method described in [ 13 ]. The masks were manually analyzed based on statistics on number of lungs found and size ratio between right and left lung. Only the masks with anomalies on these statistics were visualized. The code used to segment the patients was adapted for the cases with unsatisfactory segmentation. After this, all patients with anomalies presented a satisfactory mask.

Pathological changes in lungs a ected by tuberculosis may be represented by a large variety of ndings. In most cases such nding include aggregations of foci and in ltrations of di erent sizes. However, rarer types of lesions may be present including brosis, atelectasis, pneumathorax, etc. "Left lung a ected" and "Right lung a ected" labels provided with the CTR data set indicated presence of any kind of TB-associated lesions in the left and right lung, respectively. Typical examples of CT ndings are shown in Fig. 1. Pleurisy, calci cations, caverns and lung capacity decrease were considered separately from the other types of lesions. Pleurisy is known as in ammation of the membranes that surround the lungs and line the chest cavity7. Calci cations are usually represented by densely cal cied foci that look like bright spots (usually more than 1000 Houns eld Units) on CT images [ 14 ]. Calci cations may occur inside of lungs but also can be located on vessels and the mediastinum. Caverns, also known as pulmonary cavities, are gas- lled areas of the lung in the center of nodules or areas of consolidation [ 15 ]. Lung capacity decreased indicates the decrease of volume of the a ected lungs compared to normal lungs. Lung capacity de7 https://www.nhlbi.nih.gov/health-topics/pleurisy-and-other-pleural-disorders creased can be caused by many factors and can be often associated with other CT ndings such as pleurisy and presence of large caverns. 2.3

Evaluation Measures and Scenario

Similar to the previous editions, the participants were allowed to submit up to 10 runs to each of the two subtasks. In the case of the SVR task, the participants had to provide the probability of HIGH severity for each patient. During the challenge, this task was evaluated with area under the receiver operating characteristic (ROC) curve (AUC) and accuracy and the runs were ranked rst by AUC and then by accuracy. Moreover, we included the unbalanced Cohen kappa coe cient to our analysis and the ROC curves are provided in Section 3.

In the case of the CTR task, the participants had to provide the probability of each CT nding (see Section 2.1) for each patient, i.e. for each patient they had to provide a 6-dimensional vector with the probabilities. This task was considered a multi-binary classi cation problem and standard binary classi cation metrics are provided. During the challenge the runs were ranked based on the average AUC and the min AUC obtained. In addition, since the data set was highly unbalanced for some of the CT ndings (see Table 1), we include the AUC, sensitivity and speci city for each nding. 2.4

Participation

In 2019 there were 97 registered teams and 48 signed the end user agreement. 13 groups from 11 countries participated in one or more subtasks and submitted results. These numbers are similar to 2017 and 2018, where there were 90 registered teams, 50 that signed the end user agreement, and 10 teams from 9 countries submitting results. Table 2 shows the list of participants and the subtasks where they participated. 3

Results

This section provides the results obtained by the participants in each of the subtasks. 3.1

SVR Subtask

Table 3 shows the AUC and accuracy obtained by each participant's run, measures used to establish the SVR ranking. The ROC curve for the best run of each participant is shown in Figure 2. In addition, Table 4 summarizes the results for each best run and includes the unweighted Cohen Kappa coe cient. The best results were obtained by the UIIP BioMed [ 16 ] group, both in terms of AUC and accuracy. The same group also ranked rst in the previous edition, obtaining a signi cant improvement this year: from 0.7025 to 0.7877 AUC. For this edition, they proposed an initial convolutional neural network (CNN) using 2D projections of the 3D CT scans that provides a probability of high TB severity. Then they combined these probabilities with the available meta-data and used a linear regression classi er to provide the nal classi cation score. The UIIP [ 17 ] group obtained the best Kappa. In their approach, they rst performed data augmentation and used a 3D CNN as autoencoder, followed by a traditional classi er, such as random forest.

A total of ve groups (including UIIP BioMed) participated in both editions of this subtask (2018 and 2019), and all obtained higher results in 2019: HHU [ 18 ] improved from 0.6484 to 0.7695 AUC. They proposed a completely new approach where they rst assessed the CT- ndings proposed in the CTR subtask and then applied linear regression to obtain the severity score. In addition, they also tried a di erent approach based on selecting 16 CT slices and using a 3D CNN (UNet) that obtained lower results. The SD VA HCS/UCSD [ 19 ] used an ensemble of 2D CNNs, combining the predictive scores provided by each CNN. Then they fuse these scores with the meta-data into a Support Vectors Machine (SVM) classi er that provided the nal severity score. With this approach they went from 0.6658 to 0.7214 AUC. The performance of MedGIFT [ 20 ] remained almost the same between both editions (0.7162 vs 0.7196 AUC). Their best approach in 2019 is similar to the one proposed in 2018. They proposed to model the lung as a graph by dividing the lung elds into a number of subregions (di erent for each patient) and considering these subregions as nodes of a graph. They then de ned weighted edges between adjacent subregions, where the weights encode the distance between 3D texture descriptors obtained in each subregion (node). In order to compare the obtained graphs, they transform these graphs into a lung descriptor vector and used SVM to classify them. In addition, they also attempted to classify the graphs with a 2D CNN obtaining much lower results. Finally, the last group participating in both editions is the MostaganemFSEI [ 21 ] group, that improved from 0.5987 to 0.6510 AUC. Their pipeline consisted on rst selecting meaningful axial CT slices manually. These slices are then described with semantic features extracted via a 2D CNN. As a last step, they used a 5-class long short term memory (LSTM) algorithm to obtain one of the original 5 levels of TB severity that is then transformed into the classes high or low.

CompElecEngCU [ 22 ] created 2D derived images by concatenating sagittal and coronal CT slices that are classi ed with a hybrid of a 2D CNN based on AlexNet and a Multi-Layer Perceptron. The UniversityAlicante [ 23 ] group considered each CT volume as a time series (or video) and used optical ow on the 3 directions. The SSN CoE [ 24 ] and UoAP [ 25 ] groups used a similar approach. Both rst manually selected a set of relevant slices for each patient and then used a CNN. In the case of SSN CoE they selected 30 slices and used a 2D CNN. UoAP used a 3D CNN (VoxNet) with either 16 or 32 CT slices. Finally, the FIIAugt [ 26 ] group performed random sampling of pixels of the CT volumes and used a combination of decision trees and weak classi ers. 3.2

CTR Subtask

To provide a ranking in this subtask we used the mean AUC and min AUC over the six binary CT- ndings proposed. Table 5 provides these two measures for all runs submitted. Similar to the SVR subtask we provide more detailed results for the best run of each group. For each best run and for each CT- nding, Figures 3, 4, 5 and 6 depicts the ROC curves, AUC, sensitivity and speci city, respectively. In this case, the sensitivity and speci city metrics have been computed assuming the standard decision threshold of 0.50. Moreover, Table 6 summarizes the results of the best runs providing mean, min and max values for each of these metrics.

Again, UIIP BioMed [ 16 ] is the winner of this subtask with a mean AUC of 0.7968 and a min AUC of 0.6860. When we check the individual AUCs for each CT- nding (see Figure 4), we observe that they outperformed every other method in the left and right lung labels by a high margin. However, they have similar results to other techniques in the other four CT- ndings. In this subtask they used di erent approaches for each CT- nding, mainly consisting of a unique 2D CNN architecture with modi ed input for each abnormality. It is worth to mention their simple technique for detecting pleurisity: they noticed that most of the lung masks provided by the organizers did not contain the areas of the lungs presenting pleurisity. Therefore, they used their own lung segmentation algorithm based on atlas registration. The nal score for pleurisity was then computed based on the di erence between their masks and the organizer's masks. HHU [ 18 ] is the other group that used a speci c method for each CT- nding, mainly based on morphological operations and binarizations with a standard classi er as a last step. In the case of the MostaganemFSEI [ 21 ] modi ed the last step of the pipeline applied in the SVR subtask, substituting the LSTM step with an SVM classi er. PwC [27] and LIST only participated in this subtask. The latter did not provide details of their approach. In the case of the PwC group they used 3D CNN with 20 slices for feature extraction and used them along with the meta-data in a random forest classi er. All the other groups participating in this subtask, i.e. CompElecEngCU, MedGIFT, SD VA HCS/UCSD, UIIP and UniversityAlicante, used the same approach (or with minor modi cations) than in the SVR subtask. 4

Discussion and Conclusions

In the second edition of the SVR subtask, we observe a signi cant improvement by most of the groups that participated in both editions. However, since we transformed the original 5-class regression task into a binary classi cation problem, AUC is the only metric that we can compare between editions. The nal results, around 0.80 AUC and 0.70 accuracy, encourage us to continuing investigating the task. Most of the participants used the clinical meta-data provided, but unfortunately we cannot analyze the individual contribution of these data.

Left lung a ected

Right lung a ected Lung capacity decrease

Presence of calci cations Presence of pleurisy

Presence of caverns

The results obtained in this rst edition of the CTR subtask showed impressive performance by the participants. Already combining only the best runs analyzed in this work, the AUC for each CT- nding would be of 0.8796, 0.9254, 0.8360, 0.7554, 0.8467 and 0.7955, respectively. However, the sensitivity and speci city seem to not correlate with the AUCs obtained. This is due to the lack of optimization of the classi cation decision threshold ( xed to 0.50 in our analysis) and this also explains the inverse behavior between speci city and sensitivity of most of the methods. Since the ranking in the CTR task was announced to be evaluated only by AUC, adjusting the decision threshold was not required and hence we assume that no participant adapted the predictions. At the same time, this suggests that maybe AUC was not the best metric to evaluate/rank the methods. A priori, it seems that the AUC only provided information about whether the methods of the participants were capable of ordering the predictions, i.e. that a patient with an abnormality presents higher positive probability than a patient without it, but this does not assure that the method is able to distinFig. 5. Sensitivity (true positive rate) obtained by the best run of each group for each CT nding. The dashed line marks the sensitivity of a random classi er, 0.50. guish between presence and absence of a certain CT- nding. Something worth mentioning is the misalignment between the training and test sets in terms of the proportion of positive patients in some of the CT- ndings, e.g. lung capacity decrease (29-7%) and presence of calci cations (13-51%) (see Table 1). Preserving the proportions for all the CT- ndings simultaneously was a extremely di cult task due to the relative small size of the data set. We believe that this misalignment interfered with the generalization power of some methods.

The participants developed many di erent approaches in 2019, with many of them applying deep learning (DL) techniques. This is actually representative of the current trends in the medical imaging community where DL methods are gaining terrain in almost every area. However, some of the preliminary analysis performed on the CT images by the participants proved that it is more important to understand the problem than to have powerful methods. These analysis led to simple approaches for some of the abnormalities that resulted in high performance (e.g. assessing the presence of pleurisity by comparing lung segmentation masks). We also noticed that all participants model the CTR subtask as a multibinary problem, with few groups adding a relation between the abnormalities. This was expected since the dataset was not large enough to model the CTR subtask as a multi-label problem due to the high variability when having six labels to predict. Nonetheless, we nd surprising that only few groups used their predictions of the CT- ndings in their assessment of the TB severity score.

Overall, the 2019 edition of the ImageCLEF TB task again proved the high interest by the medical imaging community in this task resulting in the highest participation of the three editions. Moreover, the results once again support the bene ts of applying machine learning techniques in the assessment of a TB severity score, and more precisely in the detection of TB-associated abnormalities. The use of a unique data set for the two tasks allowed to provide a rich set of meta-data for all the patients that was used by most of the participants. However, providing such meta-data a ected the size of the data set. In future editions of this task we will focus on extending the data set without reducing the amount of meta-data provided.

Acknowledgements

This work was partly supported by the Swiss National Science Foundation in the project PH4D (320030{146804) and by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-18-64818-1 "Year 7: Belarus TB Database and TB Portals". ceedings., Lugano, Switzerland, CEUR-WS.org <http://ceur-ws.org/Vol-2380> (September 9-12 2019) 27. Pattnaik, A., Kanodia, S., Chowdhury, R., Mohanty, S.: Predicting Tuberculosis Related Lung Deformities from CT Scan Images Using 3D CNN. In: CLEF2019 Working Notes. Volume 2380 of CEUR Workshop Proceedings., Lugano, Switzerland, CEUR-WS.org <http://ceur-ws.org/Vol-2380> (September 9-12 2019)

References

1. Kalpathy-Cramer , J. , Garc a Seco de Herrera, A. , Demner-Fushman , D. , Antani , S. , Bedrick , S. , Muller, H.: Evaluating performance of biomedical image retrieval systems: Overview of the medical image retrieval task at ImageCLEF 2004{2014 . Computerized Medical Imaging and Graphics 39 ( 0 ) ( 2015 ) 55 { 61

2. Muller, H., Clough , P. , Deselaers , T. , Caputo , B., eds.: ImageCLEF { Experimental Evaluation in Visual Information Retrieval . Volume 32 of The Springer International Series On Information Retrieval. Springer, Berlin Heidelberg ( 2010 )

3. Garc a Seco de Herrera, A. , Schaer , R. , Bromuri , S. , Muller, H.: Overview of the ImageCLEF 2016 medical task . In: Working Notes of CLEF 2016 ( Cross Language Evaluation Forum) . (September 2016 )

4. Muller, H., Clough , P. , Hersh , W. , Geissbuhler , A. : ImageCLEF 2004 {2005: Results experiences and new ideas for image retrieval evaluation . In: International Conference on Content{Based Multimedia Indexing (CBMI 2005 ), Riga, Latvia, IEEE ( June 2005 )

5. Ionescu , B. , Muller, H., Peteri , R. , Dicente Cid , Y. , Liauchuk , V. , Kovalev , V. , Klimuk , D. , Tarasau , A. , Abacha , A.B. , Hasan , S.A. , Datla , V. , Liu , J. , DemnerFushman , D. , Dang-Nguyen , D.T. , Piras , L. , Riegler , M. , Tran , M.T. , Lux , M. , Gurrin , C. , Pelka , O. , Friedrich , C.M. , de Herrera , A.G.S. , Garcia , N. , Kavallieratou , E. , del Blanco , C.R. , Rodr guez, C.C., Vasillopoulos , N. , Karampidis , K. , Chamberlain , J. , Clark , A. , Campello , A. : ImageCLEF 2019: Multimedia retrieval in medicine, lifelogging, security and nature . In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Volume 2380 of Proceedings of the 10th International Conference of the CLEF Association (CLEF 2019 )., Lugano , Switzerland, LNCS Lecture Notes in Computer Science , Springer (September 9-12 2019 )

6. Ionescu , B. , Muller, H., Villegas , M., de Herrera , A.G.S. , Eickho , C. , Andrearczyk , V. , Dicente Cid , Y. , Liauchuk , V. , Kovalev , V. , Hasan , S.A. , Ling , Y. , Farri , O. , Liu , J. , Lungren , M. , Dang-Nguyen , D.T. , Piras , L. , Riegler , M. , Zhou , L. , Lux , M. , Gurrin , C. : Overview of ImageCLEF 2018: Challenges, datasets and evaluation. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018 ), Avignon, France, LNCS Lecture Notes in Computer Science , Springer (September 10 -14 2018 )

7. Ionescu , B. , Muller, H., Villegas , M. , Arenas , H. , Boato , G. , Dang-Nguyen , D.T. , Dicente Cid , Y. , Eickho , C. , Garcia Seco de Herrera , A. , Gurrin , C. , Islam , B. , Kovalev , V. , Liauchuk , V. , Mothe , J. , Piras , L. , Riegler , M. , Schwall , I. : Overview of ImageCLEF 2017: Information extraction from images . In: Experimental IR Meets Multilinguality, Multimodality, and Interaction 8th International Conference of the CLEF Association , CLEF 2017 . Volume 10456 of Lecture Notes in Computer Science., Dublin, Ireland, Springer (September 11 -14 2017 )

8. Villegas , M. , Muller, H., Garcia Seco de Herrera, A. , Schaer , R. , Bromuri , S. , Gilbert , A. , Piras , L. , Wang , J. , Yan , F. , Ramisa , A. , Dellandrea , A. , Gaizauskas , R. , Mikolajczyk , K. , Puigcerver , J. , Toselli , A.H. , Sanchez , J.A. , Vidal , E.: General overview of ImageCLEF at the CLEF 2016 labs . In: CLEF 2016 Proceedings. Lecture Notes in Computer Science , Evora. Portugal, Springer ( September 2016 )

9. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Garc a Seco de Herrera, A. , Bromuri , S. , Amin , M.A. , Kazi Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , Roldan Garc a , M.d.M.: General overview of ImageCLEF at the CLEF 2015 labs . In: Working Notes of CLEF 2015. Lecture Notes in Computer Science . Springer International Publishing ( 2015 )

10. Caputo , B. , Muller, H., Thomee , B. , Villegas , M. , Paredes , R. , Zellhofer , D. , Goeau , H. , Joly , A. , Bonnet , P. , Martinez Gomez , J. , Garcia Varea , I. , Cazorla , C. : ImageCLEF 2013: the vision, the data and the open challenges . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

11. World Health Organization, et al.: Global tuberculosis report 2016 . ( 2016 )

12. Dicente Cid , Y. , Kalinovsky , A. , Liauchuk , V. , Kovalev , V. , , Muller, H.: Overview of ImageCLEFtuberculosis 2017 - predicting tuberculosis type and drug resistances . In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings , Dublin, Ireland, CEUR-WS.org <http://ceur-ws. org> (September 11-14 2017 )

13. Dicente Cid , Y. , Jimenez-del- Toro , O. , Depeursinge , A. , Muller, H.: E cient and fully automatic segmentation of the lungs in CT volumes . In Orcun Goksel, Jimenez-del- Toro , O. , Foncubierta-Rodriguez , A. , Muller, H., eds. : Proceedings of the VISCERAL Challenge at ISBI. Number 1390 in CEUR Workshop Proceedings (Apr 2015 ) 31 { 35

14. Bendayan , D. , Barziv , Y. , Kramer , M. : Pulmonary calci cations: a review . Respiratory medicine 94(3) ( 2000 ) 190 { 3

15. Gadkowski , L.B. , Stout , J.E. : Cavitary pulmonary disease . Clinical Microbiology Reviews 21 ( 2 ) ( 2008 ) 305 { 333

16. Liauchuk , V.: ImageCLEF 2019: Projection-based CT Image Analysis for TB Severity Scoring and CT Report Generation . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

17. Kazlouski , S.: ImageCLEF 2019: CT Image Analysis for TB Severity Scoring and CT Report Generation using Autoencoded Image Features . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

18. Bogomasov , K. , Braun , D. , Burbach , A. , Himmelspach , L. , Conrad , S. : Feature and Deep Learning Based Approaches for Automatic Report Generation and Severity Scoring of Lung Tuberculosis from CT Images . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

19. Gentili , A. : ImageCLEF2019: Tuberculosis - Severity Scoring and CT Report with Neural Networks, Transfer Learning and Ensembling . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

20. Dicente Cid , Y. , Muller, H.: Lung Graph-Model Classi cation with SVM and CNN for Tuberculosis Severity Assessment and Automatic CT Report Generation . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

21. Hamadi , A. , Cheikh , N.B. , Zouatine , Y. , Menad , S.M.B. : ImageCLEF 2019: Deep Learning for Tuberculosis CT Image Analysis . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

22. Mossa , A.A. , Yibre , A.M. , Cevik , U.: Multi-View CNN with MLP for Diagnosing Tuberculosis Patients Using CT Scans and Clinically Relevant Metadata . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

23. Llopis , F. , Fuster , A. , Azor n , J., Llopis , I. : Using improved optical ow model to detect Tuberculosis . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws.org/Vol2380> ( September 9-12 2019 )

24. Kavitha , S. , Nandhinee , P. , Harshana , S. , Jahnavi Srividya , S. , Harrinei , K. : ImageCLEF 2019: A 2D Convolutional Neural Network Approach for Severity Scoring of Lung Tuberculosis using CT Images . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

25. Zunair , H. , Rahman , A. , Mohammed , N.: Estimating Severity from CT Scans of Tuberculosis Patients using 3D Convolutional Nets and Slice Selection . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Proceedings., Lugano , Switzerland, CEUR-WS.org <http://ceur-ws. org/ Vol- 2380 > ( September 9-12 2019 )

26. Tabarcea , A. , Rosca , V. , Iftene , A. : ImageCLEFmed Tuberculosis 2019 : Predicting CT Scans Severity Scores using Stage-Wise Boosting in Low-Resource Environments . In: CLEF2019 Working Notes . Volume 2380 of CEUR Workshop Pro-