-
The goal of the ImageCLEFmed 2020 Tubercolosis task3 [
8, 6
] is to detect
whether the different parts of the lung are affected by Mycobacterium
tuberculosis. The categories are LeftLungAffected, RightLungAffected, CavernsLeft,
CavernsRight, PleurisyLeft, PleurisyRight. The data set contain 403 computed
tomography scans (CT scans). Out of the 403 CT scans 283 scans are used as
a training set with known labels for the participants and 120 CT scans as the
test for the competition. For our experiments we split the training set into two
subsets (163 as training and 120 as validation set) and evaluated our models on
the smaller set. Out of the two lung masks [
2, 10
] we used the first segmentation
method in the aggregation phase of the slice predictions.
Models
First, we preprocessed individual CT slices. Based on the provided lung masks
we centralized and rescaled the slices to lower the resolution from 512x512 to
256x256. Additionally we standard normalized the intensity values, see Fig. 1. We
omitted to apply additional augmentation techniques [
11
] e.g. rotation, mirroring
or random crop as the position of the lung is crucial. We treated the scoring
procedure as a set of binary classification tasks therefore we trained separate
neural networks for each category.
We chose feed-forward neural networks with a single output node to model
the categories per slice. Every inner layer included Rectangular Linear Units
(ReLU) as non-linear activation functions and we chose sigmoid for the output
unit. We built a traditional Convolutional Neural Network (CNN [
9
]) with two
convolutional layers with 64 5x5 sized filters and a Residual Network (ResNet
[
5
]) with three residual blocks, for details see Table 1 and Table 2 respectively.
The residual blocks contained a set of 3x3 sized convolution with 8,32,64 filters
per block followed by a second convolution with the same size and a final
residual connection and a downsizing unit. Between the two convolutions we used
batch normalization and ReLU similarly to the original paper. Before the linear
discriminative layer we downsized the tensor with average pooling.
Additionally, for the CNN network we applied Dropout [
11
] in the second convolutional
layer. We evaluated the performance of the models with the log-likelihood of
the probability of the original label measured by the activation of the output
unit. As an optimization method we used Adam [
7
] thus we omitted additional
regularization in the loss function.
We measured the performance of various models on the validation set, a
random subset of the training set. In the testing phase we used every training
scan with the best settings. We implemented the models in PyTorch framework
4 and did all the experiments in python. Additionally, we used the provided lung
masks based on the first automatic segmentation method described in [
2
].
2.1
Aggregation
We combined the individual predictions of the slices to compute a single score
per CT scan. During our experiments we applied various methods to define a
single score:
– Mean score (sc1): mean of the individual prediction scores of the CT scan.
– Maximal score (sc2): maximal prediction score in a CT scan.
– Minimal score (sc3): minimal prediction score in a CT scan.
– Median score (sc4): median prediction score in a CT scan.
– Middle score (sc5): prediction score of the center slice.
– Majority vote (sc6): proportion of the positive predictions.
4 https://pytorch.org
– Mask score (sc7): weighted prediction scores based on the proportion of the
actual lung in the slices. The proportion of the lung was the proportion
of the lung segment given the mask files. The masks were extracted by a
fully automatic lung segmentation method described in [
2
]. We used the
corresponding masks per lung per task.
All of our submitted runs included mean score over the CNN results. Our main
submission (#68061) achieved mean AUC of 0.595. The remaining runs
contained single category scores with random scores for the rest of the categories
(we estimated the AUC as AU Cmean ∗ 6 − 0.5 ∗ 5)). The estimated per category
AUC of our submission can be seen in Table 3. We noticed that two of the
categories achieved an AUC under 0.5 thus if we negate the scores the AUC
values will flip to the upper half and the adjusted mean AUC will be 0.6548.
Important to mention, that these adjustments only provide us information about
Layer
Input layer
Residual layer 1
Residual layer 2
Residual layer 3
Average pooling 8x8
Output layer
the distinguishing capability of the models (how the model differentiate
negative and positive examples), in a realistic scenario the final decisions would be
still wrong as without any test data we would not know that we need to flip
the scores. During the challenge and afterwards we experimented over the small
training (163 CT scans) and validation (120 CT scans) sets with several models
and aggregation methods. Table 4 show the mean AUC results on the
validation set and the detailed AUC scores can be seen for the left and right lung in
Table 5 and in Table 6 respectively. The method (mean score of CNN) in our
main submission achieved a mean AUC 0.595 on the validation set however the
best method (median score of CNN) performed significantly better with AUC
of 0.642. If we select the best model (ResNet or CNN) with the median score
per category the mean AUC will be similar to the median CNN with 0.659. In
comparison, if we select properly both the model and the aggregation method
for each category the mean AUC increases to 0.686, a significant gain on the
validation set in comparison to the submitted run.
In this paper we described our submission and some additional experiments
over the data set of the ImageCLEFmed 2020 Tuberculosis task. We trained
traditional feed-forward convolutional and residual neural networks over the
individual slices of the CT scans and combined the predictions based on the
importance of the slices according to their position and how well they represent both
of the lungs. We found that median score performed best on average although
in some categories the middle slice score or the mask score outperformed other
aggregation methods. Both ResNet and traditional CNN performed similarly in
our experiments on the validation set while the residual network needed
significantly higher computational power. Our simplest run which was submitted to
the challenge had very low mean AUC score 0.595 meanwhile with additional
aggregations we improved the same method on the validation set to achieve a
mean AUC 0.684. We plan to replace 2D convolutions with 3D convolutions to
take advantage of the complex structure of CT scans. Additionally, we intend to
further expand our experiments with bi-directional Recurrent Neural Networks
(RNN [
4
]) to read through the CT scans from both ends and classify the
sequence as a whole, utilize Markov Random Fields [
1
] over the prior predictions
and generate additional samples with slice transition refinement with inter-slice
reconstruction and with category-wise Generative Adversarial Networks [
3
] to
boost the training procedure. Based on the submissions of other participants
(SenticLab.UAIC mean AUC 0.924 or SDVA-UCSD mean AUC 0.875) we
believe individual slice predictions may not be representative enough to describe
CT scans as a whole to detect Mycobacterium tuberculosis.
5