Early detection of tuberculosis can save many lives as it remains one of the leading causes of death, half a century after its discovery. The analysis of chest CT scanned images can be a quick and economic mechanism for detecting not only the type of tuberculosis, but also differentiating whether or not the disease is multi-drug resistant. These are two of the objectives of the ImageClef Tuberculosis task of 2018, and are the ones studied by the group of the University of Alicante in this edition. We have carried out two work approaches, one based exclusively on the use of Deep Learning techniques on a sequence of 2D images extracted from a 3D tomography and on a second approach using Optical Flow to convert the 3D tomography into a motion representation in order to calculate the ADV (a previous descriptor provided by the group). This descriptor is able to synthesize the information of a sequence into one image. This article presents the experiments carried out and the results obtained within the task.
ImageClefTuberculosis is one of the tasks of ImageClef 2018 [11]. The
ImageCLEFtuberculosis task 2018 [
In this rst participation our initial objective was to compare two models, Deep learning and Optical Flow to check their results in task 1. Finally we made a delivery about task 2 using the second model that had given us better results in the experimentation of the task 1.
This document is structured as follows: in sections 2 we present the architectures of the models used: Deep Learning and Optical Flow. In section 3 we show the experimetntation done with both models. Section 4 presents the o cial results of the experiments and Section 5 summarizes the document and o ers a series of proposals for future work. 2 2.1
Our approaches to the solution
Deep neural networks have managed to solve problems or increase e ciency in
problems related to image processing [
To address the issue of the rst task (resistant form of tuberculosis), 3D chest images of CT scan are used. In a rst stage, these 3D images are transformed into a sequence of 2D images each one that represent the entry of the neural networks.
Input: multichannel image
Conv1
1. Convolutional neural network (CNN) with data augmentation: The main idea is to use the advantages of convolutional layers for a single multi-channel image. In this case, each channel would be a 2D gray image (see Figure 1). 2. Convolutional layers combined with a recurrent neural network: The natural way to combine the advantages of convolutional layers and sequential Convolutional modules
LSTM l a iton rs lvou lyae n o C e ltt a s a n tIr e n h1 l a iton rs lvou lyae n o C e ltt a s a n tIr e n h2 ... ... ...
l a iton rs lvuo lyea n o C e ltt a s a n tIr e n hn r e y a l e s n e D t u p t u O
treatment is to combine it in networks with multiple inputs per tomography.
Figure 2 shows a basic scheme about this approach.
3. Pretrained network and classi cation: As rst approximation to extract
features from an image VGG16 deep convolutional neural network [16] is used
with the ImageNet [12] weights learned (4096 features per image). The main
idea is concatenate the features of each input image belonging to the same
tomography to get the nal features vector to classify in a classical way.
4. Pretrained network and classi cation as a sequence using recurrent neural
network. In this case, the extraction of features is similar to the one described
in the previous paragraph and each feature vector would be considered as
a component of a sequence to be treated by a well knowns recurrent neural
network called Long-Shot Term Memory (LSTM) [
In this sections we propose a combined method based on optical ow and a characterization method called ADV, to deal with the classi cation of chest CT scan images a ected by di erent types of tuberculosis. The key point of this method is the interpretation of the set of cross-sectional chest images provided by CT scan, not as a volume but as a sequence of video images. We can extract movement descriptors capable of classifying tuberculosis a ections by analyzing deformations or movements produced in these video sequences.
The concept of optical ow refers to the estimation of displacements of
intensity patterns. This concept has been extensively used in computer vision in
OF-MIADV. Optical Flow plus MIADV
In this sections we propose a combined method based on optical flow and a characterization
method called MIADV, to deal with the classification of chest CT scan images affected by
different types of tuberculosis. The key point of this method is the interpretation of the set of
cross-sectional chest images provided by CT scan, not as a volume but as a sequence of video
dimi ageerse.nWteacpanpleixctaratcitomnovdeommenatindes:scrrioptboorstcaoprabvleehoficclaessnifyaivngigtautbieorcnu,locsiasraffdercitvioinnsgb,y video
saunravlyesiinllgadnecfoermoratfioancsiaolr mexopvermesesnitosnpro[6d]u.ceInd inbitohemseedviidceaolsecqounetnecxets. optical ow has been
uThseedcontocepatnoafloypztiecalofrlogwa nrefderesftoortmheaetsiotimnsation of dispWlaceemcaennts onfidntednisity pattermns.eTthhisods in
[
n
n
transformation cross-sectional chest
r 5 d f l u ADV 3x3x5
n
r 5 d f l u
The figure summarizes thFeisgu.c3ce.ssive stages of the process forperxotrcaecstsingsttahgeeasctivity descriptors
Optical ow plus ADV (optical flow+ADV) that will be the input of a classifier. In the first stage a transformation over the cross-sectional chest images provided by the CT scan is performed in order to transform imagTehfeormgautsr eintsou mvidmeoarsiezqeusetnhcees saudcacpetesdsivtoe csatlacugleasteoofptthiceal pflroowc.eTsshefosreceoxntdrastcatgieng the aimctpilvemiteyntdsetshceriLputcaosrsKa(noapdteicmaelthoodwt+oAobDtaVin)otphticaatl wFloilwl.bTehet htheirdinsptaugte ocfalaculcaltaesssdie er. In the rst stage a transformation over the cross-sectional chest images provided by the CT scan is performed in order to transform image formats into video sequences adapted to calculate optical ow. The second stage implements the Lucas Kanade method to obtain optical Flow. The third stage calculates the activity description vector ADV (3x3x5) accumulating within each 3x3 region of the image, the displacements of the optical ow in four directions of a 2D space (right, left, up, down). The fth component of the ADV calculates the frequencies in direction changes. In the fourth stage a normalization of the ADV vector in performed. Finally, the last stage uses the ADV vector normalized as the input for a generic classi er in order to evaluate the results. 3 3.1
Experimentation
Preliminary experiments using Deep Learning
In order to validate the results the wide 10-fold cross validation (10-CV)
technique are used and 7 images of 2D are extracted from the original 3D
tomography. For the experiments Keras v2.1.6 [
Table 1 shows the rst approach using CNN. The results are close to 50% which means that the network has not learned the di erence between the two classes.
The second approach consists of combination of CNN with RNN. In this case, 2 layers are used and the lters are: 32 (3x3), 64 (3x3). The accuracy is 0.50 and individual proofs are [0.54 0.58 0.42 0.50 0.48 0.52 0.52 0.44 0.48 0.52]. The results are also unsatisfactory and we will try a new approach.
Conv. Detail layers lters x kernel
Accuracy mean 10-CV results
The third try using a pretrained network (VGG16) with ImageNet weights con guration. In this case, VGG16 is used to extract the weights of the penultimate layer as descriptors of image. These features extracted from the latest layers of the neural network are called neural codes. The number of nal characteristics is 28672 corresponding to 7 images per times 4096 neural codes per image. Table 2 summarize the experiments using classi ers belonging di erent families of algorithms attending to neural codes directly or normalizing with L2 function.
Last approach using deep learning architectures consists of get neural codes as previous try and classify the sequence of 7 images with a recurrent neural network (LSTM). Again, the accuracy is 0.49 and detailed fold results are [0.62 0.54 0.42 0.46 0.28 0.48 0.48 0.48 0.56 0.56].
In general, the results per folder (10-CV) are very di erent probably due to the nature of neuronal networks with random initialization of neurons, the optimizers that have to adjust thousands of parameters that nally nd local minimums and also due to the small amount of images available to train a neuronal network where small di erences between the training and test sets Neural Classi er Codes algorithms
Nearest Neighbors Linear SVM RBF SVM
Decision Tree original Random Forest
AdaBoost Naive Bayes Logistic Regression XGBoost
Accuracy 10-CV mean results allow generating sets easier to classify in some cases than in others. On the other hand, no preprocessing has been applied to 2D images which could also in uence the high variations in results. 3.2
Preliminary experiments using Optical Flow plus ADV For this experiments, the wide 10-fold cross validation (10-CV) technique have been used again. All images of the original 3D tomography are used to calculate the optical ow for each patient. For the experiments Matlab R2013b has been used to calculate the optical ow, the ADV and the classi ers.
Table 3.2 shows the performance results of the proposed method.
Classi er OF size ADV Accuracy MDR Accuracy DS Accuracy SVM 64x64 3x3 0,5097 0,312 3-knn 64x64 3x3 0,5135 0,52
Table 3. Classi cation results using Optical ow plus ADV 0,6567 0,4627 3.3
Frequency Matrix with Deep Learning A modi cation of the Optical Flow experiment was to use the frequency matrices generated as input to a neural network.
In gure 1 you can see an example of Frequency Matrix.. 4
Results 1. Run 1: MDR Baseline. The Baseline is a probabilistic model in which the image was not analyzed and only the data of sex and age have been taken into account. 2. Run 2: ADV 3x3, SVM, 1000 SMOTE upsampling
As can be see in the table 4 the model of Optical Flow SVM obtains the best results, for the sake of using only selected images.
Due to we had little time available for second task, we only present the two models of Optical Flow, SVM and 3nn.
Run 1: ADV 3x3, SVM, 1000 SMOTE upsampling Run 2: ADV 3x3, 3-nn, 1000 SMOTE upsampling
The results were signi cantly better using the 3-nn but very far from the rest of the participants 5.
Conclusions and future work Early detection of tuberculosis is a major social challenge, given the devastating e ects of the disease. On the other hand, it represents a scienti c challenge of the highest level. As the organizers claim, \you have to work to get methods that allow a correct detection of the disease that kills thousands and thousands of people". In this paper we have proposed two di erent approaches to face the problem. The rst one is based on the use of Deep Learning techniques on a sequence of 2D images extracted from a 3D tomography. The second approach uses Optical Flow to convert the 3D tomography into a motion representation in order to calculate the ADV (a previous descriptor provided by the group). This descriptor is able to synthesize the information of a sequence into one image. The experiments carried out in these two approaches allow us to con rm the interest of these lines of research and encourage us to seek improvements in the proposed methodologies. 11. Ionescu, B., Muller, H., Villegas, M., de Herrera, A.G.S., Eickho , C., Andrearczyk, V., Cid, Y.D., Liauchuk, V., Kovalev, V., Hasan, S.A., Ling, Y., Farri, O., Liu, J., Lungren, M., Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., Gurrin, C.: Overview of ImageCLEF 2018: Challenges, datasets and evaluation. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018), vol. 11018. LNCS Lecture Notes in Computer Science, Springer, Avignon, France (September 10-14 2018) 12. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classi cation with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097{1105 (2012) 13. Patel, D., Saurahb, U.: Optical ow measurement using Lucas Kanade method.
Int J Comput Appl 61(10), 6{10 (2013) 14. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825{2830 (2011) 15. Rumelhart, D., Hinton, G., Williams, R.: Learning sequential structure in simple recurrent networks. Parallel distributed processing: Experiments in the microstructure of cognition 1 (1986) 16. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) 17. Xavier, M., Lalande, A., Walker, P.M., Brunotte, F., Legrand, L.: An adapted optical ow algorithm for robust quanti cation of cardiac wall motion from standard cine-MR examinations. IEEE Transactions on Information Technology in Biomedicine 16(5), 859{868 (2012). https://doi.org/10.1109/TITB.2012.2204893