The paper presents and comparatively analyses several deep learning approaches to automatically detect tuberculosis related lesions in lung CTs, in the context of the ImageClef 2020 Tuberculosis task. Three classes of methods, di erent with respect to the way the volumetric data is given as input to neural network-based classi ers are discussed and evaluated. All these come with a rich experimental analysis comprising a variety of neural network architectures, various segmentation algorithms and data augmentation schemes. The reported work belongs to the SenticLab.UAIC team, which obtained the best results in the competition.
1;2 Medical imaging technologies like Computer Tomography (CT) and Magnetic Resonance (MR) produce high volumes of data in the form of volumetric images. The richness of information they provide is essential to correct diagnosis but brings at the same time new challenges, both for manual/human and automatic/machine processing: these are not only about the size of the produced data but also about the complexity of the diagnosis process itself. With respect to automated diagnosis, the volumetric images, which can be seen both as matrices of pixels/voxels or series of 2D images (usually called slices), produced high e ervescence in the deep learning research community, triggering a variety of new architectures and approaches.
The current paper makes use of deep learning to automatically detect
tuberculosis and related a ections in lung CTs, in the context of the ImageClef
Tuberculosis task [
The paper is structured as follows. Section 2 describes the challenge and the
dataset. Section 3 describes the approaches developed based on reducing the
volumetric image to 2D projections, starting with the previous year winning
approach reported in [
The challenge in the 2020 ImageClef Tuberculosis competition is the automatic detection of tuberculosis and related lesion types in CTs. The CT report to be generated must contain 3 binary labels for each lung, indicating the presence of TB lesions in general, the presence of pleurisy and caverns in particular.
The training dataset consists of 283 CTs. All CTs present at least one lung affected, 19 have pleurisy and 126 caverns. Because we split each CT into left/right lungs, this translates to 566 inputs, 444 a ected, 21 with pleurisy and 145 with caverns.
The task is therefore a multi-binary classi cation problem, with three target labels per lung. For each target label the AUC is computed and the ranking is done on a test set, rst by computing the average AUC and then by the minimum AUC over the 3 target labels.
We split the data into train/validation in the same fashion as [
The 3D matrix representing the volumetric image can be reduced to simpler 2D
representations by traversing it in each of its three dimensions and computing
statistics on numeric vectors. In the case of lungs CTs, a segmentation algorithm
is rstly applied to detect the lungs and eliminate the other parts in the CT.
Further, we used the method proposed in [
The rst step behind all our approaches is image segmentation, with the aim
of identifying and isolating each lung in the volumetric image. Because the
performance of further processing is greatly in uenced by the quality of
segmentation (especially in the case of the 2D projection approach where the projections
take into account the entire volume), we tested several segmentation methods.
The organizers provided for all patients two versions of automatically extracted
masks of the lungs: one which relies only on anatomical assumptions [
Feeding a VGG neural network [
Consequently, all our further experiments use the segmentation provided by
non-rigid registration [
Fig. 1. Projections dataset creation ow using 4 segmentation variants (in order: growth-based, registration-based, unetLTRCLobes and unetR231). Although the 4 types of projections resulted look only slightly di erent, the di erence in classi cation scores is signi cant. 3.2
The images go through a series of augmentations, each with a certain probability of being applied, including: horizontal and vertical ipping, small degrees of rotations, blurring, added gaussian noise, distortions, random cropping, and changing di erent values of hue, saturation or brightness. We used the Albumentations 5 library for most of these augmentations. 3.3
In an e ort to improve over the last year result, we used the pre-processing
provided in [
steps that the authors mention, except the regional maxima calculation. Figure 2 illustrates the di erence between projections with and without further preprocessing.
For training we chose AlexNet[
We further tried di erent variants of Resnet[
We extracted the lungs using the registration-based segmentations, computed
all 3 projections, split by lung side and processed with the augmentation we
listed in section 3.2. We trained the networks and aggregated the results on
all 3 projections and computed the mean score to obtain the nal results back
at CT level. We tried di erent approaches in aggregating the results including
training a small neural network, but found the simpler mean aggregation to
give the highest score. We thus obtained our highest score in the 2D approach
using a resnet-50 network [
In an attempt to make use of the whole volume at once, we used SqueezeNet in a 3D version, based on the implementation found in the repository 6. In order to work with volumes of di erent sizes, we used batch size equal to 1. To handle volumes with a large number of slices, we used Apex7 library for reducing the burden on our GPU. In a preliminary experiment we considered only the case a ected vs. not a ected. We noticed the bad results during the training: after some epochs the prediction scores stagnated, for all volumes, between 0:4 and 0:6. We concluded that 3D convolutions are not able to capture the important information on our small training set of volumetric images. 4.2
In our attempt to associate the entire volumetric image to a label, we constructed a hybrid approach. We fed the volume slice by slice into a convolutional neural network, fused the resulted feature maps at channel level and continued with another small convolutional network into a prediction. The initial convolutional neural network is composed from the encoder part of a U-net[14] architecture which was pretrained on a segmentation task at the end of which we applied a squeeze connection to reduce the number of channels, fused the resulting feature maps so that the slices processed in parallel by the CNN would now be treated as channels of a single input, then applied a resnet-like small network to compile the features into a label. We no longer use the masks to extract the lungs but instead use a simple threshold based segmentation to compute the boundaries of the body and crop out the space around the it. We again split by lung side and used only horizontal ip as a preprocessing. The resulting volume is resized to
the xed size of (128, 256, 256) The network could then be fed images in batches multiple of 128 representing the slices of a volume.
To make maximum use of the GPU memory, we used the Apex library to train using mixed precision, in a distributed manner on 2 GPUs. We could t 2 times 128 images into the memory corresponding to 2 volumes.
The approach turned out to be cumbersome. The time to process an epoch was relatively high and the convergence of the network seemed slow. After 2 days of training we decided to stop and the network reached an AUC of around 0.6 on the hold-out validation set. 5
Having a closer look at the training set, one can observe that usually the lesions on the lungs are located only on a small number of slices from the whole volume. A natural idea is to try a 2D model that could di erentiate between healthy lung slices and lung slices with lesions (caverns, pleurisy and a ections) and construct the CT report based on the ndings at slice level. For this purpose we need training data labeled at slice level and not CT level.
The rst approach was to try to automatically detect the slices presenting lesions in the training set, using a lung nodule detector8 constructed by the winners of a challenge in cancerous nodules detection. The results were bad, the model not being able to recognize the slices showing caverns although these correspond to big, obvious regions.
Therefore, we started to manually select from each volume of the training set the slices with lesions. We actually found that this was not as time-consuming as we initially thought, by processing only the volumes labeled with lesions, and it de nitely was worth the e ort, as the increase in performance shows. The caverns are usually big and obvious and the a ections are either nodules or dusty lungs images (which may indicate pneumonia), with very rare cases of pneumotorax (the lung disappearing due to the outbreak of a cavern). There are many cases when these lesions appear on a very small number of slices and thus, the two approaches described in sections 3 and 4 might have not been able to reveal them. 5.1
In our rst tries using the slice by slice approach, we used InceptionNet version 3 [15]. We used the annotated data in di erent ways. Transforming each slice of a volume into a picture, resizing each image to a size of 299 299, cutting the picture in half to obtain the two lungs and using vertical ip for all the pictures which are either a ected, with caverns or with pleurisy, are the data pre-processing steps for our rst attempt using this approach. This approach
does not use the provided segmentation masks at all. We only used 4 labels as output: a ected, caverns, ok, pleurisy.
As input for the neural network we used several versions, having all images as 3-channel images:
a) NaiveInception. We added a linear layer on top of the last adaptive average pooling layer of the architecture, keeping the original InceptionNet weights freezed. With this approach we scored 0.86 AUC score on the test set, surpassing this way the best approach based on 2D projections.
b)ThresholdInception. The other approaches consist in using some other preprocessing steps. This time, when creating the photos from the 3D volume, we used a Window Width and Window Level equal to 1500, -500 respectively. This way we improved the results to 0.887 mean AUC and to 0.82 min AUC on the test set.
c) TwoPicInception. One other approach consists in mimicking the protocol a doctor has to follow in order to decide a ections(including caverns) and pleurisy. In order to see the a ections more clearly, a doctor uses Window Width and Window Level equal to 1500, -500 respectively, whereas for better visualisation of pleurisy, a doctor looks at pictures with Window Width and Window Level equal to 350, 50 respectively. With this thresholding, the liquid surrounding the pleura becomes more observable. Figure 4 top shows the di erences between the two pictures.
In order to use information from 2 pictures during training, we used two InceptionNet modules, with trainable parameters and concatenated the two outputs of the adaptive average pooling layer. The decision was made based on the output of the last linear layer applied on the concatenation discussed above. With this approach we scored 0.89 mean AUC on the test set.
d) AttentionInception.We wanted to gain insight into how accurate the methods can be. We tried to check the predictions produced by the methods proposed and discovered that the models found in a big proportion correct slices of the volumes which contained certain a ections. In order to work on the explainability of our model, we modi ed the structure of ThresholdInception, using the idea from [16], introducing an attention mechanism. Instead of feeding the output of the last pooling layer into the linear layer, we used dot product attention [17]. We computed similarity scores between the output of the pooling layer and three di erent convolutional layers in the architecture. After using the compatibility scores as weights for the features extracted by the three layers, we concatenated the new features. Using a linear layer on top, we predicted scores for the 4 categories. After plotting the attention we noticed that the attention on the rst layer selected highlighted the whole lung area - supporting the idea that we don't need the segmentation masks, whereas the second layer of attention highlighted a ections on the lungs. With this approach we scored 0.85 mean AUC. We believe this lower performance is due to the fact that the attention on the last layer was not good. Checking the visualisation for that layer we noticed useless areas highlighted (Fig. 4, bottom). Because of the limit imposed on the number of submissions, we stopped investigating this direction.
For all the methods above based on InceptionNet, training was performed for 30 epochs on one GPU Nvidia RTX 2070 with 8GB of memory, using Stochastic gradient descent optimizer. We divided the learning rate at each 10 epochs by 10 and used binary cross-entropy as loss function.
In order to establish the diagnosis for a volume we applied the following heuristic: we applied the inference step on all the pictures/slices from the volume and for each of the possible classes we took the maximum score encountered; if only one slice was found with an a ection score higher than 0.8, then we divided the score of a ected by 2. 5.2
E
cientNet
In an e ort to use a powerful, yet small footprint network, in our last approaches
we used e cientnet[18], speci cally the b4 variant which has only 19M
parameters but reaches top 1 accuracy of 82,6% on Imagenet. We used a Pytorch
implementation pre-trained on Imagenet[
The preprocessing we used here is similar to the ones we used before and took place at run-time on load. We applied the registration-based mask per slice based on a threshold to crop the body and remove much of the surrounding space, split the lungs into left/right (just by using splitting the image in half) and applied the same series of augmentations as in the previous approaches. The split and cropped image has dimension 256 256 and after randomly cropping it reduces to 224 224. For ease of working we also kept the size of the volumetric image depth to a xed 128 slices per volume.
If otherwise speci ed for this approaches as for the others we used a window level of -500 and range of 1500 corresponding to the most common values used in areas of acute di ering attenuation values (example: lungs) where air and vessels will sit side by side.
As input we tried several options, all of them maintaining 3 channels per image:
a) Micro-volumes (MicroVolSlice): The importance of volumetric data is evident. This seemed especially apparent when we try to manually identify caverns which can present as rounded or irregularly shaped black centers surrounded by a white contoure. The caverns can range in size from small with a thin contoure line to large with thick and di use borders. The small caverns we found especially hard to identify as it can be confused with a section of a larger blood vessel. As untrained individuals, to eliminate the confusion we traced the potential cavern a few slices up or down to verify if it continues into a vessel or forms a pathology. To try and mitigate this type of confusion in a model we composed the 3 channels of the image from 3 consecutive (or equidistant) slices. In case the slice is at the beginning or end of the sequence we simply duplicated it to ll the channels.
b) False-color (FalseColorlSlice): To make use of the entire range of values of an MRI image we established 3 intervals in the Houns eld units range to correspond to the 3 channels of an image. The window size and level used 9 are (1500, -500) corresponding to the usual values used for lung imaging, (350, 40) called narrow window (used when examining areas of similar attenuation, for example, soft tissue) and (500, -600) a narrower window of the usual values for lung imaging in an attempt to retain more information around the values corresponding to blood vessels and soft tissues. The result with this method however was not submitted to the site as the result on the hold-out set was poorer than the others.
Fig. 6. The schematic of the slice approach (microvolumes). It follows a simple ow. The volumetric image, split by left/right side is cropped using a simple threshold based segmentation, then composed (in this case) to microvolumes, augmented and passed through the model. The output from all the slices of a side of a volume is then aggregated and the max per label selected to compose the nal result for a side. c) Naive (NaivelSlice): The image is simply duplicated into the 3 channels.
To compile the nal results for each volume we aggregate the individual results per slice and choose the max of each each label across all the slides, which are then used to compute the AUC.
Loss: Cross entropy vs Binary cross entropy : A strange case comes from the fact that using the CrossEntropyLoss (on a multi-label classi cation) without softmax before the loss (thus assigning a predominant label for each slice) we obtained higher results than using BinaryCrossEntropyLoss. The nature of the results is also very di erent, the rst giving results on the extremes while the latter hovering around 0.5, but both giving decent results around 90% AUC.
Micro-volumes and just repeating the image gave similar results on the test set (92.2% and 92.4% respectively), however the training on the "naive" case was done on 130 epochs on 3 GPUs with batch 56 3 whereas the "microvolumes" case was done on 60 epochs on 2 GPUs with batch 56 2. Therefore, the approach with the highest score on our hold out set was c) the simple one which also represented the highest of the submitted scores. Our second highest submitted model is represented by the approach in a) micro-volumes.
For training we used Nvidia RTX 2070 with 8GB of memory. We again, used the Apex library from NVidia to train using mixed precision and DistributedDataParallel with one process per GPU. 6
Table 1 summarizes the results obtained in the competition on the test set. Submissions were made only for the methods based on 2D projections and the ones based on predictions at slice level; as shown on the hold-out validation data, the attempts to use the whole volume using 3D convolutions or fusing the information at slice level did not obtain good results and therefore were not used in the competition test phase. By comparing the results both on our hold-out validation set and on the test set, the following conclusions can be drawn.
{ As indicated by the low training accuracy, the approaches using the entire volumetric data as a whole corresponding to the segmented lungs (described in section 4), involving 3D convolutions or slice fusion, seem to be overwhelmed by the amount of parameters to t and are not able to identify the lesions in cases where these are small or present only on a few slices of the CT, or either converge slowly. { The 2D approaches based on projections computed over the segmented volume (described in section 3) give (unreasonable) good results, which indicates that simple (normalized) statistics like mean, maximum and standard deviation, when used together, are able to catch important information about the presence of lesions in lung CTs. The quality of segmentation of the lungs is of critical importance in this case, as a bad segmentation may introduce noise into the projections. After obtaining the set of 2D projections, data augmentation increased the generalization capability of the classi er. { The best approach, surpassing signi cantly the ones based on 2D projections, exploits all the information present in the segmented volumetric lungs in a slice-wise manner. Instead of predicting the presence of the a ection per CT, we predict it for each slice. To obtain the report back at lung level the probabilities over slices are aggregated by extracting the maximum. An important pre-processing step consisted in xing the window and range levels to speci c values used by radiologists when inspecting lung CTs. 7
Volumetric images like CTs and MRIs provide rich information about the internal body structure, necessary in the diagnosis of many a ections. With the advancements of neural networks, automatic diagnosis in volumetric images became possible at high precision, useful for prioritizing patients and assisting doctors in nal decisions. After a thorough experimental analysis of various architectures, the current paper devised an approach able to produce highly accurate CT reports about the presence of tuberculosis related a ections. The method, based on computing predictions at slice level, has, beside high accuracy in predicting lesion type, the advantage of o ering more information in terms of localization of the lesions. With a current mean AUC score of 0.924 on test data, its performance can be increased if more data, capturing various cases, is provided in the training phase. 8
Our strong belief is that data science without domain knowledge can never reach its full potential. We would like to thank Mirela Iordache, an outstanding radiologist at the Regional Institute of Oncology in Iasi, for sharing her knowledge and valuable insights on the medical niche approached in the competition. 14. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234{241. Springer, 2015. 15. Christian Szegedy, Vincent Vanhoucke, Sergey Io e, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 16. Saumya Jetley, Nicholas A Lord, Namhoon Lee, and Philip HS Torr. Learn to pay attention. arXiv preprint arXiv:1804.02391, 2018. 17. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. 18. Mingxing Tan and Quoc V Le. E cientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, 2019.