In 2018, tuberculosis was one of the top 10 causes of death worldwide. Especially patients that develop a multidrug-resistance are endangered and need special medical treatment. Within the ImageCLEF 2018 challenge the automatic distinction between drug-sensitive and multidrug-resistant tuberculosis was investigated by only using the CT scan, age and gender of a patient. In this paper, we present di erent approaches using convolutional neural networks, decision trees and the combination of both classi ers. We show that our models achieve competitive results regarding the other participants of the challenge and show an improvement with respect to our last year's results. All of them are represented in the ranks between 4th and 26th of 39. Our best method regarding the AUC measure reached a score of 0:5810. In regard of accuracy the best approach got a result of 0:572.
Tuberculosis is a disease that is caused by an infection with the mycobacterium tuberculosis. Although the bacterium was found about 135 years ago, it still is one of the top ten causes of death worldwide according to the World Health Organization (WHO)1. Due to medical progress the disease can be cured. Some patients, however, can develop a resistance to several drugs. This circumstance complicates the medical treatment and must therefore be recognised as soon as possible. Since the distinction between drug-sensitive (DS) and multidrugresistant (MDR) tuberculosis is di cult and necessitates several expensive tests, it would be helpful to nd an automated solution that only requires the CT scans that are usually done anyway.
This year, the ImageCLEF 2018 tuberculosis Task 1 [
In contrast to the last year's challenge, additionally to the images the age and gender of the patients were given. Also the number of images was increased. Nevertheless, the amount of training data is still relatively small and therefore another challenging factor.
In this paper we introduce our approaches for the task which include the usage of convolutional neural networks, decision trees and the combination of both classi ers. 2
In the ImageCLEF 2017 tuberculosis challenge the same task was set for the
rst time[
The second best team used a combination of convolutional and recurrent neural
networks and achieved results between 0:562 and 0:5501 on the ranks 4 { 6. The
model was published in [
Regarding the accuracy, however, our team aimed the rst rank with 0:5681
using convolutional neural networks with a at network architecture [
Since the manual distinction between DS and MDR tuberculosis using only CT Scans is not possible until now, it is very di cult to determine relevant features of the images. For this reason, the usage of neural networks, which do not require feature extraction, is reasonable. On the one hand a convolutional neural network can be used, which only considers the images, on the other hand an architecture can be developed, which examines the images as well as the additional features age and gender. The combination of the images and the textual features cannot only be done by a CNN with multiple inputs, but also by creating a feature vector of the CNN's result, the age and the gender, and using it as the input of another classi er. In this work, a decision tree has been chosen as the second classi er. Before discussing the di erent approaches, the preprocessing of the (1a) (2a) (1b) (2b) images, which has a great impact on the results, will be explained, since it is the same for all models. 3.1
The CT scans are three-dimensional grayscale images whose intensity values
are speci ed by Houns eld Units (HU) [
The rst approach of this work is a classi cation of the images using a
convolutional neural network, that takes the three dimensional scans as input. This was
done by using the Keras API [
Since the usual SPP was made for 2D images and the used CNNs were three
dimensional, a few modi cations had to be made. These were already carried
out and described last year in [
8 n.a. n.a.
(4, 4, 1) categorical cross entropy and the adam optimizer. Both networks can handle an un xed input size because of the SPP. Nevertheless, the nets have been trained and tested with images of di erent as well as uniform sizes.
In Table 2 the architecture of the second type of CNNs is shown. Instead of the SPP, the atten layer is used. Besides, one additional block of a maxpooling and convolution layer is inserted. The di erence between the structure of Flatten and Flatten3 is in the second and third block, where Flatten performs the convolution only over the X- and Y-axis of the images while Flatten3 considers all axes. Both networks use binary cross entropy and stochastic gradient descent. Another submitted architecture is called FlattenX. It is very similar to Flatten.
Age &
Fig. 2: Architecture of the CNN with multiple inputs.
Instead of the rst maxpooling layer, however, it has a convolutional layer with 8 lters and a lter size of (5; 5; 3). The intention of this modi cation was to avoid a possible loss of information in the rst layer.
The last submitted CNN is the MultiInputCNN, which takes an image and a two dimensional vector as inputs. As can be seen in Figure 2, the rst part of the net is identical to the Flatten CNN. After the atten layer, however, a dense layer with 200 units is used. Its result is merged with the second input which contains the age and gender of the considered patient. This data is then processed by a dense layer with the size of 2. 3.3
As already mentioned in the introduction of Section 3, all given information can be combined by using multiple classi ers. To re ne the results of the CNN, a decision tree was trained using the CNN's binarized output, the age and the gender. The decision tree was chosen because the considered features and decisions are easy to track. This is very helpful to understand the impact of CNN's results.
Depending on the selection of the network the classi er learned whether to
consider the CNN's result or not. In Figure 3 the structures of the two best decision
trees are illustrated. On the right, the result of using the Flatten net is
displayed. After checking age and gender the CNN's output is considered, as well.
In contrast to this, the results of the Flatten3 net have no in uence on the
decision tree's assignment at all. The structure is shown on the left of Figure 3.
It is noticeable that the tree in a) exactly corresponds to the rst part of the
one in b). The models have been retrieved by using the DecisionTreeClassi er
from Scikit-Learn [
y not resistant
Age < 30?
Gender = 1?
y (a) n Age < 55? n resistant n not resistant resistant y
Age < 30? y
y Gender = 1? n Age < 55? n resistant n not resistant
CNN < 0.5 y
n Age < 43?
not resistant y not resistant n resistant (b)
Fig. 3: Illustration of the two best decision trees' structures. fraction per leaf was set to 10%, the minimum impurity decrease was 0:01. All architectures used the Gini Impurity as a measure for the information gain. 4
The training set that is provided by the ImageCLEF 2018 [
When using a xed input size, all images have then been resized to a X- and Y-length of 250 pixels and a total number of 100 slices. If the length of a side (x, y or z) was too small, it was enlarged by adding equally black borders on both sides of the axis. If the length was too big, interpolation was used to downsize the side with the biggest variance regarding the desired dimension, so that the proportions of the side lengths remained the same. Afterwards the sides that became too small were enlarged as described.
All networks have been trained with the complete training set and for at least 30 epochs. In the combination of CNN and decision tree, the network rst had been trained with 180 and the tree with 40 randomly selected CT scans which had an even type distribution. Afterwards the network was post-trained with all given training images.
In Table 3 the preliminary results of the challenge are displayed as published
on the website2. For a better understanding a few les of our team have been
renamed. As our results are among the top 20 regarding the AUC score, only an
excerpt of the top 25 of 39 runs was selected. The combination of the Flatten3
CNN and the decision tree achieved the best AUC as well as Accuracy score
regarding the complete set of our submitted runs. This method was ranked
6th with respect to the AUC measure and 4th in regard of the Accuracy. The
run is named "MDR Flatten3 DTree.txt". "MDR Flatten DTree.txt" represents
the predictions of the combined approach with the Flatten CNN. It is noticeable
that both runs retrieve exactly the same results. That is, because the predictions
are the same, as well. This phenomenon can be explained by the fact that the
network classi ed all scans of patients of gender 1 and the age between 43 and 55
as not resistant. The additional branch of the tree (in regard to the architecture
without CNN) has therefore never been reached using the test set.
Just behind our best runs we have the results of Conv68 which was trained
with images of a xed size. The networks Conv48, Flatten and Flatten3 reached
with similar results the AUC ranks 11, 12 and 13. Regarding the accuracy, the
Flatten architecture achieved the second best rank for our team. Unexpectedly,
2 http://www.imageclef.org/2018/tuberculosis
date: 28.06.18
the MultiInputCNN received the worst results. The achieved accuracy of 0:5593
is the same for the ranks 9 to 12. The FlattenX network reached the second
worst AUC results for our team. This con rms the thesis from [
We have shown that our approaches achieve competitive results. Despite the small amount of data, the use of convolutional neural networks can be reasonable if the architecture is not too deep. All our runs reached AUC ranks between 6 and 20 and accuracy ranks between 4 and 26 of 39. The best classi er turned out to be the decision tree, which only takes into account the age and gender of the patient. Its AUC and Accuracy results are only about 4% worse than the best scores of the challenge. Nevertheless, it has to be said that the accuracy results are worse than those of a classi er that only categorizes into the class with the most representatives. This one would reach a score of 58:05%.
The quality of the classi ers can certainly be increased by improving the preprocessing of the images with the help of medical expertise. Furthermore, the optimization of the provided masks could lead to better results, because these do not consider relevant regions of the lungs and contain parts of bones in some cases. Also, it would be interesting to perform the classi cation with other known medical data besides the age and gender of the patients.