ImageCLEF 2020 Tuberculosis Task is an example of the challenging research problem in the field of CT image analysis. The purpose of this research is to make accurate estimates for the three labels (affected, pleurisy, caverns) for each of the lungs. We describe the tuberculosis task and approach for chest CT image analysis, then perform multi-label CT image analysis using the task dataset. We propose finetuning deep neural network model that uses inputs from multiple CNN features. In addition, this paper presents two approaches for applying mask data to the extracted 2D image data and for extracting a set of 2D projection images along multi-axis based on the 3D chest CT data. Our submissions on the task test dataset reached a mean AUC value of about 75% and a minimum AUC value of about 69% Copyright ⃝c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2020, 22-25 September 2020, Thessaloniki, Greece.
With the spread of various virus (such as Tuberculosis, Coronavirus, and
Influenza), medical researchers perform to give the necessary treatment for viruses
in recent years. However, there is nothing to identify the disease early. Early
diagnosis needed to give the necessary treatment, develop specific medicine, and
prevent the death of patients. Therefore, several researchers have invested their
efforts in recent years, especially within the medical image analysis community.
In fact, a task dedicated to the tuberculosis had been adopted as part of the
ImageCLEF evaluation campaign in its editions of the four last years [
In this paper, we also employ a new fine-tuning neural network model which uses features coming from pre-trained CNN models as input. In addition, existing deep learning MODELS had weak classifications, therefore we propose a new fully connected 2 layers. The new contributions of this paper is to propose a novel feature building techniques, which incorporates features from two CNN models to predict Tuberculosis from images, unlike most recent research only concerned with adopting single CNN features. In the following, we first describe the tasks which were conducted in Section 2 followed by dataset of ImageCLEF2020, In Section 3, we introduce masking the dataset, experimental settings, and feature used in this research . In Section 4, we describe experiments we have carried out. In Section 5 we conclude this paper . 2
The tuberculosis task of ImageCLEF 2020 Challenge included part of chest in
format of 3D CT images [
This task participants have to generate automatic lung-wise reports based on CT image data. Each report should include the probability scores (ranging from 0 to 1) for each of the three labels and for each of the lungs (resulting in 6 entries per CT). The resulting list of entries includes: LeftLungAffected, RightLungAffected, CavernsLeft, CavernsRight, PleurisyLeft, PleurisyRight. Table 1 shows labels for the chest CT scan in the Training dataset.
In Training set
We propose a multi-label analysis system to predict Tuberculosis from CT scan images. The first step is the input data pre-processing. After pre-processing input data, we will describe our deep neural network model that enables the multilabel outputs, given CT scan images. In addition, we add an optional step to the first step. We use a CT scan movie not CT scan images. We will detail our proposed system in the following section. 3.1
First, we remind the reader that in train and test data, 3D CT scans are provided in compressed Nifti format. We decompressed the files and extracted the slices of x-axis, y-axis, and z-axis from the three dimensions of the 3D image shown in Fig. 1. For each dimension for each Nifti image, we obtained a number of slices ranging according to the dimension: 512 images for the X and Y dimensions, and from 110 to 250 images for the Z dimension.
After extracting slices along x-axis, y-axis, and z-axis, we propose to filter
the slices of each patient using mask data [
To solve our multi-label problem, we propose new combined neural network models which allow inputs coming from End-to-end (CNN) features. Training and Validation sets The training dataset consists of 108 891, 77 468, 31 497 images extracted from the filtered CT image for x, y and z axis respectively.
We have divided the train data into training and validation data with 8:2 ratio in random. CNN features were extracted using pre-trained CNN-based neural networks, including VGG16, ResNet50, NasNet-Large and EfficientNet B07. In order to deal with the above feature, we propose a deep neural network architecture where we allow multiple inputs and a multi-hot vector output.
Our system incorporates CNN features, which can be extracted using deep
convolutional neural networks pre-trained on the ImageNet [
AUC for four fine-tuning the CNN model from above. As illustrated in Fig. 3, CNN feature is combined and represented by an integrated feature as a linearly weighted average, where weights are w3 for CNN features, respectively. CNN feature is passed out on “Fusion” processing to generate the integrated features, followed by “softmax” activation function. 3.3
We propose a method illustrated in Algorithm 1. The input is a collection of features extracted from each image with K kinds of sentiments, while the output is a K-dimensional multi-hot vector.
In Algorithm 1, we assume that the extracted CNN feature is represented by their probabilities. For each Tuberculosis, we sum up the features, followed by median of the result, which is denoted by Tik in Algorithm 1. In short, the vector Si represents the output multi-hot vector. We repeat this computation until all the test (unknown) images are processed.
The train dataset consists in filtering CT image on x-axis, y-axis, or z-axis. The train dataset consists of 108 891, 77 468, 31 497 images extracted from the filtered CT image for x, y and z axis respectively.
Algorithm 1 Predicting multi hot vector for an image Input: Image data i including K kinds of disease for Lungs Output: Multi hot vector Si 1: for k do range (K): 2: P robi,k=F eatureExtractioni,k 3: Tik=median(P robi,k) 4: end for
Here, we have divided the filtering data into training and validation data with 8:2 ratio. We determined the following hyper-parameters; batch size as 256, optimization function as “SGD” with a learning rate of 0.001 and momentum 0.9, and the number of epochs 200. For the implementation, we employ TensorFlow[14] as our deep learning framework. For the evaluation of multi-label classification, we employ mean Area Under Curve (AUC). Table 2 shows the results. Here we compare in terms of AUC for multiple axes. For fine-tuning EfficientNet B07 in x, y, and z-axis, it turns out that our proposed CNN model has the best AUC. Finally, we employ EfficientNet B07 for training and validation sets and test data. The result shows as below (4.2). axis x-axis y-axis z-axis
Model VGG16
ResNet50
NasNet-Large EfficientNet B07
VGG16
ResNet50
NasNet-Large EfficientNet B07
VGG16
ResNet50
NasNet-Large EfficientNet B07
The test dataset consists of 46 605, 32 901, 13 938 images extracted from the filtered CT image for x, y and z axis respectively.
We expect that our proposed models could give better results after a more advanced data preprocessing including the use of filtering image, and data augmentation for multi-axis. Here, we described above, we employ fine-tuning CNN models in EfficientNet B07 based on multi axis. Table 3 shows the results. “xaxis and y-axis” mean the probabilities of x-axis and y-axis. “y-axis and z-axis” mean the probabilities of y-axis and z-axis. “x-axis, y-axis, and z-axis” mean the probabilities of x-axis, y-axis, and z-axis.
Here we compare in terms of AUC. For z-axis on fine-tuning EfficientNet B07, it turns out that our proposed CNN model has the good mean AUC and minimum AUC.
In addition, results of the participants’ submissions with the highest AUC
values are shown in Table 4 [
In this research, we proposed a model for predicting each of the three labels and for each of the lungs as a multi-label problem from chest CT images. We performed Chest CT Image analysis where we proposed a combined deep neural network model which enabled inputs to come from CNN features. In multilabel Chest CT Image analysis, we also introduced a threshold-based multi-label prediction algorithm. Specifically, after training our deep neural network, we could predict the existence of a disease for given unknown CT scan images. Experimental results demonstrate that all our proposed models outperform the individual pre-trained CNN model in terms of mean AUC and minimum AUC.
In this research, we proposed a model for Tuberculosis CT Image analysis which accurately estimates multi-label problems from given images. The multilabel problems are evoking multiple different types of Tuberculosis findings simultaneously.
In the future, given an arbitrary CT or X-ray image might be included the optimal weights for the neural networks. Moreover, we hope our proposed model can encourage further research on the early detection of several viruses or unknown diseases. We also expect that our proposed model will be widely used in the field of medical computing.
A part of this research was carried out with the support of the Grant-in-Aid for Scientific Research (B) (issue number 17H01746), and Grant for Education and Research in Toyohashi University of Technology. 12. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. 13. Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML 2019, 05 2019. 14. Google. Tensorflow. https://github.com/tensorflow.