In this article, we present the methodologies used in our participation in the two subtasks of the ImageCLEF 2019 Tuberculosis Task (SVR and CTR). Our contributions are essentially based on deep learning and other machine learning techniques. In addition to the use of deep learners, semantic descriptors are tested to represent patients CT scans. These features are extracted after a rst learning step. Our submissions on the test corpus reached AUC value of about 65% in the SVR task and an average AUC value of about 63% in CTR. These results o ered us the seventh and the eighth places in SVR and CTR, respectively. We believe that our contributions could be further improved and might give better results if they applied properly and in an optimized way.
Tuberculosis (TB) is a deadly disease. Its early diagnosis can give the necessary
treatment and prevent the death of patients. The technological advancement
especially in the eld of arti cial intelligence and precisely supervised learning
opens the door for researchers to study the possibility of an automatic diagnosis.
This would speed up the process and lower its cost. Several researchers have
invested their e orts in recent years, especially within the medical image analysis
community. In fact, a task dedicated to this disease had been adopted as part of
the ImageCLEF evaluation campaign in its editions of the three last years [
We present in the following section our work that had been made in the
context of our participation in the two sub-tasks of ImageCLEF 2019
Tuberculosis task: Tuberculosis Severity Scoring (SVR) and Tuberculosis CT Report
(CTR) [
The remainder of this article is organized as follows. Section 2 describes the two tasks in which we had participated. In section 3, we present our contribution by detailing the system deployed to perform our submissions. Section 4 details our experimental protocols we used to generate our predictions. We present and analyze in the same section the results obtained. We make our conclusions in the last section by presenting potential perspectives and future works. 2
Participation to ImageCLEF 2019
ImageCLEF 2019 [
In this paper, we focus on our participation in the SVR and the CTR sub-tasks. The main objective of these two challenges is the automatic analysis of Tuberculosis CT scans. In both tasks, the same dataset is used, one corpus for training and another one for testing. The data is provided as 3D CT scans. All the CT images are stored in NIFTI le format with \nii.gz" extension le (gzipped .nii les). For each of the three dimensions of the CT image, we nd a number of slices varying according to the dimension considered (512 images for the Y and X dimensions, from 40 to 250 images for the Z dimension). Each slice has a size of about 512 128 pixels for the X and Y dimensions and 512 512 pixels for the Z dimension.
A training collection is provided at the beginning of the task with its groundtruth (labels of samples). Participants prepare and train their systems on this dataset. A test collection is provided at a later date. Participants interrogate their system and submit their predictions to the organizers' committee. An evaluation is performed by the latter to compare the performance of the participants' predictions submissions.
SVR task aims to predict the degree of severity of TB cases. Given a CT scan of TB patient, the main goal is to predict the severity of his illness based on his 3D CT scan. The degree of severity is modeled according to 5 discrete values: from 1 (\critical/very bad") to 5 (\very good"). The score value is simpli ed so that values 1, 2 and 3 correspond to \high severity" class, and values 4 and 5 correspond to \low severity".
The classi cation problem is evaluated using two measures: 1) Area Under ROC-curve (AUC) and 2) Accuracy.
CT Report task has as objective to automatically generate a report based on the patient's CT scan. This report should include the following six pieces of information in the binary form (0 or 1): 1. Is the left lung a ected? 2. Is the right lung a ected? 3. The presence of calci cations; 4. The presence of caverns; 5. The presence of pleurisy; 6. The lung capacity decrease.
This task is considered as a multilabel classi cation problem (6 binary ndings). The ranking of this task is done rst by average AUC and then by min AUC (both over the 6 CT ndings).
We proposed to use the system presented in Figure 1. The latter goes through two essential steps: input data pre-processing and training a classi cation model. A third optional step is added in order to improve the performance of the rst learning step. The latter includes a second learning stage by using a recurrent neural network (LSTM) or by generating semantic features and exploiting them through a learner or a deep learner. We will detail our proposed system in the following.
of slices
descriptors labels Slices
We remind that in both tasks, 3D CT scans are provided in compressed Nifti format. Firstly, we decompressed the les and extracted the slices. In the end, we got three sets of slices corresponding to the three dimensions of the 3D image. For each dimension and for each Nifti image we obtained a number of slices ranging according to the dimension considered (512 images for the Y and X dimensions, from 40 to 250 images for the Z dimension).
The visual content of the images that were extracted from the di erent dimensions is not similar. Indeed, the images of each dimension are taken from a di erent angle of view. We noticed from our experiments that the slices of the -Z- dimension give better results compared to the two others (X and Y). This remark concerns our proposed approaches. This is why we used in our work the Z-dimension. However, all steps can be applied to slices of any of the three dimensions.
After choosing the dimension to consider, we propose to lter the slices of each patient. Indeed, we can notice that many slices do not necessarily contain relevant information that could help to classify the samples. This is why we added a step to lter and select a number of slices per patient. For this, we propose two ltering approaches:
Automatic supervised ltering: In this approach, we select a set of
patients from each of the considered classes (the ve degrees of severity for the
SVR task). Then, a professional radiologist selects for each patient, the slices
likely to contain relevant information indicating the presence of Tuberculosis.
The resulting set of slices constitutes a ltering group. Given a new patient, we
compare each of its slices to the ltering group by calculating a distance measure:
a weighted sum of distances between the slice and those of the ltering group.
This comparison can be done through a direct pixel-wise comparison. In our
experiments we used the \Structural Similarity" as distance [
CT scans Nifti format (.nii.gz)
Selecting a dimension
Z
Extracting
slices Converting Nifti to JPG
Image slices 3 Dimensions X
Y
Z
Filtering slices using; Automatic supervised filtering
or Automatic unsupervised filtering
N selected slices per
CT image / patient
Automatic unsupervised ltering: We noticed that there is usually a maximum of 50/60 slices visually informative. Since the slices are ordered, the most informative slices are usually at the center of the list. We propose then to keep only the N middle ones. This is not optimal but we opted for this choice for a fully automatic and unsupervised approach. This choice can be improved by performing manual ltering with the intervention of a human expert, preferably with medical skills on TB disease.
Deep learning model for CT image classi cation ( rst learning
step)
As a deep learner, we chose to use Resnet-50 architecture because of its good
results in the context of the same problematic in last Tuberculosis task
editions [
LungNet Deep Learner: We proposed and developed our deep learner
architecture for CT Image Analysis that we called \LungNet". The input to the
latter is an RGB image of size 119x119, followed by ve convolutional layers and
two fully connected layers. Initially, input data were in nifty format. Slices of
the CT scans are 1-channel gray-level images. However, we extracted the slices
using med2image tool [
After each convolutional layer \relu" activation is applied followed by a local normalization and MaxPooling. The rst, the second and the third convolution blocks have dropout layers to reduce over tting. The sigmoid activation function is applied to the output layer in order to predict values in the range of 0 to 1.
Figure 3 illustrates the architecture of the Lungnet model.
We implemented the method of semantic descriptors extraction described in [
So, we chose to exploit the probabilities that were predicted by a deep learner trained on a set of slices. If K is the number of classes considered, these predictions typically correspond to the K predicted probability values for the K classes (For SVR Task, K = 5: the ve severity degrees). We obtain then for each slice K values corresponding to the probabilities of the K considered classes.
Furthermore, K sub-descriptors are generated: D1, D2, D3, D4, ... Dk. Each sub-descriptor Di contains the predicted probabilities for the class i for all the slices of the patient. A nal semantic descriptor is constructed by concatenating the K sub-descriptors. Figure 4 details the whole process of the semantic features extraction for one patient. Slices
Learning a classi cation model based on semantic features
(second learning step)
We propose to exploit the semantic descriptors of patients described previously.
Any approach of supervised classi cation can be applied as shown in gure 5.
We tested in our experiments SVM as supervised classi er. However, Random
Forests and bagging of Random Forests have shown good results in the context
of the same problematic [
We recommend some ideas for this step:
{ To use a deep learner having as input the semantic descriptors of patients
and the labels of patients. As an alternative, it would be interesting to use a
bagging method that collaborates several learners and sub-samples the train
collection. This would lead to better results as presented in [
Semantic descriptors / Train corpus
Semantic descriptor 1 Semantic d.escriptor 2 . .
.
Semantic descriptor n
As each patient is described by a sequence of slices, it is interesting to test the
LSTM (Long Short-Term Memory) [
We used in our experimental work the following tools:
{ med2image [
We chose to use slices of the -Z- dimension because our experiments showed that they are more suitable than those of the two other ones and got better results.
Dataset: The dataset used in the SVR task includes chest CT scans of TB patients along with some metadata that describe a set of 19 classes. 2 classes concern the SVR task, six other classes concern CTR task. The other values are considered as additional information regarding to the patients. They could be used as contextual information. Table 1 summarizes the number of CT scans for train and test collections.
The same dataset is given for the CTR task. The samples are labeled regarding to seven main target classes: 1. Target classes for SVR Task: (a) SVR severity (binary class: HIGH and LOW). Another label called md Severity is given (Five discrete values ranging from 1 to 5). We remind that values of md Severity (1, 2 and 3) belong to the \HIGH" Severity case. The other two values (4 and 5) correspond to the \LOW" Severity. 2. Target classes for CTR Task (binary classes): (a) Left lung a ected; (b) Right lung a ected; (c) Presence of calci cations; (d) Presence of caverns; (e) Presence of pleurisy; (f) Lung capacity decrease. 4.1
Experimental protocol: We used the train collection provided by the organizers and we split it into two sub-collections: training and validation sets. We nally submitted three main runs. The other submissions concern some tested approaches that we could not optimize and nalize correctly because of lack of time: { SVR FSEI resnet50 run3: results of ResNet-50 trained on 50% of training data. Each patient was represented by 50 slices ltered using the automatic unsupervised ltering approach that was described in section 3.1. The slices were adapted by resizing them directly using the Python Imaging Library (PIL). The input images of Resnet50 are of size 199 199; { SVR FSEI lungnet run2: results of LungNet deep learner trained on 80% of data. Each patient was represented by 10 slices ltered using the automatic supervised ltering approach that was described in section 3.1; { SVR FSEI lstm run8: results of LSTM exploiting outputs of Lungnet deep learner. Each patient was represented by 50 slices ltered using the automatic unsupervised ltering approach. So, a sequence for the LSTM learner is composed of the 50 features representing the 50 slices of the patient.
We considered for each run a hierarchical classi cation problem. Firstly, we classi ed the samples in the 5 classes corresponding to the ve degrees of severity. Secondly, We deduced for each patient its predicted class using a majority vote on the predicted labels of all slices. Finally, the class predicted in the previous step is transformed to a binary value corresponding to the SVR Severity class (HIGH if predicted class 2 f1; 2; 3g and LOW if not).
Our tools and scripts used in our experiments are accessible in [
We can see that SVR FSEI resnet50 run3 got the best performance followed by SVR FSEI lstm run8. These two runs were ranked 22th and 25th out of 54 submissions.
We note that for SVR FSEI lungnet run2 patients were represented by 10 slices (50 slices for the two other runs), it would be interesting to see the performance of the Lungnet model after training it on 50 slices per patient in order to make a detailed comparison with the other two runs.
Figures 7 and 8 describe the results and ranking of all submissions of SVR task in terms of AUC and accuracy, respectively.
Although the results achieved by our submissions are not well ranked compared to those of the top of the list, we can notice that several runs belong to the same teams that had good results, and they probably do not di er too much. On the other hand, We believe that our models could give better results after a more advanced data preprocessing including the use of masks, samples selection and data augmentation.
Results of imageCLEF2019 SVR task on test collection R u U U _2
MM S s H H n _ _ e e MM
l _Mb V V
R m-S -S H H u _ _ r mm R R _ o o SV se RV RV
r r V V V F F n S S S S SR _ _ tE
R R s V V S S
A L _ R V S
mn aad sub -LRVAD -LRVAD -SSRVV -SSRVV rr_3nu -LSRVD -LSRVD lts_8n eedoC t e M R S S V S _ u n I r _ E S I_ N F E N _ S R F G
_ _ V S
R R SV SV
S R R g R
V V n V S S lu S _ 2 n u r _ I E S F _ R V S ru _ B B ss R _ _ _ E x x _V bm_D _D le V ta ta t O -a -a
a f S e e 0 Accuracy 0.4 Experimental protocol: We trained in a rst step our deep models (Resnet and Lungnet). Secondly, we generated the semantic descriptors following the approach described in section 3. We treated the problematic as a multilabel classi cation problem in the rst learning stage and as a binary classi cation problem in the second learning stage. We used in the latter SVM as a binary classi er. We optimized its parameters independently for each target class.
We submitted three main runs: 1. CTR FSEI run1: results of LungNet trained on 50% of training data. Each patient was represented by 10 slices ltered using the automatic supervised ltering approach that was described in section 3.1; 2. CTR FSEI run2 : results of LungNet trained on 70% of training data.
Each patient was represented by 50 slices ltered using the automatic unsupervised ltering approach that was described in section 3.1; 3. CTR FSEI run5: SVM using semantic features that are extracted using Resnet-50. Each patient was represented by 10 slices ltered using the automatic supervised ltering approach that was described in section 3.1.
Our tools and scripts used in our experiments are accessible in [
We can see that our best results were obtained by CTR FSEI run1
followed by CTR FSEI run2. However, we should mention here that we used
the same sub-division of the corpus in two sub-parts (train and validation) for
all CTR target classes, which is not optimal since the distribution of class
values is not the same for the six target classes. This explains the disadvantage of
the run CTR FSEI run5 compared to the other two and also the low value
of Min-AUC for the three runs. We believe that the semantic descriptors
approach might perform better by making more e orts to optimize parameters
or by testing another learner like the Bagging of Random Forests as presented
in [
Results of CTR task
We have described in this article our contributions to the SVR and CTR subtasks of ImageCLEFmed 2019 Tuberculosis task. We proposed to use after a data preprocessing step, a deep learner to classify samples to the target classes. We used for that, ResNet-50 and proposed our LungNet architecture. Moreover, we proposed to extract a single semantic descriptor for each CT image/patient instead of considering all the slices as separate samples. We tested also LSTM as another alternative. Although our proposals had not been the best, the obtained results showed that these approaches could be much more e cient and might give more interesting results if they are applied in an optimized way.
As perspectives, we plan to adopt data augmentation strategies and learning samples selection. In addition, we noticed during the sub-sampling of our data that the deletion or addition of some samples had an impact on the results. On the other hand, ltering slices in an optimized way is a key idea that could further improve the system performance. Moreover, we noticed in our experiments that there is a di erence of precision for each severity class studied which arises the hypothesis of the classes having varying di culties to be identi ed by the model. Indeed, some classes are more di cult to identify than others. It is also an interesting track to study.