A major challenge for Medical Image Retrieval (MIR) is the discovery of relationships between low-level image features (intensity, gradient, texture, etc.) and high-level semantics such as modality, anatomy or pathology. Convolutional Neural Networks (CNNs) have been shown to have an inherent ability to automatically extract hierarchical representations from raw data. Their successful application in a variety of generalised imaging tasks suggests great potential for MIR. However, a major hurdle to their deployment in the medical domain is the relative lack of robust training corpora when compared to general imaging benchmarks such as ImageNET and CIFAR. In this paper, we present the adaptation of CNNs to the sub gure classi cation subtask of the medical classi cation task at ImageCLEF 2015.
This paper documents the Biomedical Engineering and Technology (BMET)
team from the University of Sydney's submissions for the ImageCLEF 2015 [
The objective of our experiments was to evaluate the e ectiveness of
Convolutional Neural Networks (CNNs) for this subtask. In particular, we propose a
deep learning framework that could learns high-level representations of di erent
image modalities and use these to classify the modality of each sub gure.
such as image recognition, acoustic recognition and natural language processing
since 2012 [3{5]. CNNs share the common features of all deep learning
algorithms: stacked layers of neuronal subunits that learn hierarchical
representations (allowing the data to be understood at various levels of abstraction, in
isolation or combination [
A more subtle implication of deep learning is that it can automatically extract
features from raw data [3{5]. Typically, a key factor in the success of typical
machine learning algorithms is extracting salient features from the raw data.
Taking image recognition as an example, a feature set such as edges or SIFT [
Deep learning algorithms, however, are able to directly utilise raw data
instead of hand-crafted features. By feeding the data sequentially through many
successive layers of subunits, the higher levels of the system are able to
understand the data in terms of successively abstract representations [
Medical Image Retrieval (MIR) tasks, such as the tests devised for
ImageCLEF, require learning precisely these kinds of highly abstract representations,
i.e. image modality or the anatomical semantics of the image. However, to the
best of our knowledge it is not currently a well established method in this domain.
This is due to not only the inherent challenges of medical images[
Large training sets are a current necessity of very deep systems because they contain many millions of internal parameters that must be estimated from the data. Too little data can result in the the higher-level neurons' activation being the result of salient features of the training set and not re ecting the high-level representations. If this 'over tting' occurs then the system's ability to generalise on new data is severely impaired [12].
In addition to the issues regarding the volume of data required, it must be
mentioned that while deep learning can automatically perform excellent feature
extraction, this comes at the signi cant cost of the larger number of
hyperpa4 Krizhevsky et. al. [
A requirement of our classi ers was uniformly sized input vectors, however, the supplied training data varied greatly in size. This was achieved by squarecropping the image to 500px, any dimension of the image smaller than 500px was lled with black pixels.
Even prior to training the CNN, we were aware that the computational
requirements were quite demanding and this would be exacerbated by using large
images. We resized the images to 160x160px to reduce the computational
overhead that would have been required by using higher resolution images. Good
results have been reported in the literature for complex tasks with 48x48px
images [14] and Krizhevsky et. al. [
After resizing the images were 160x160x3px, the third dimension describing the three colour channels. For the purposes of simplicity and to further reduce the computational requirements we reduced the 3 channel colour representation to a single channel (red).
We randomly divided the training data into a 70/30 split for training and validation. 3.2
We evaluated the e ectiveness of CNN-derived features by comparing it to the results achieved by a Softmax classi er on the raw data. This experiment is important because the CNN's nal layer is the input to a Softmax classi er. This experiment can therefore be used to quantify the e ectiveness of the unsupervised feature extraction performed by the CNN. 3.3
The architecture for the CNN used for our experimentation was based on a simpli ed version of Yann LeCun et. al.'s [15] LeNet-55. This CNN is capable of correctly classifying the MNIST handwritten digit database with 1.7% test error. We modi ed the input to account for larger images and output a greater number of classi cations. The network consists of two convolutional pooling layers, with one fully connected hidden layer. The features that are output by the hidden layer are used for classi cation by a Softmax classi er. The architecture of the system is shown in Figure 1. The speci cations of the convolutional-pooling layers are detailed in Table 1.
As mentioned earlier the CNN requires a great deal of computational resource to run. It took approximately 3.5 hours to train a single epoch for each model, while training two models simultaneously on the CPU of a powerful system6. However, the models were not able to converge before the submission deadline. As such the runs that we submitted were based on only partially converged models. The details of the four runs submitted are detailed in Table 3. The rst Softmax model, trained for 1000 epochs, produced a 0% validation error. This was interpreted as being the result of severe over- tting to the supplied training data. Despite the fact that this classi cation scheme was essentially a baseline to evaluate the performance of CNN-extracted representations over the raw data, it was thought prudent to perform second run, with less training and hopefully less over tting, in order to see the results of a more general model. Thus, for sf run 5 we submitted the results of training the same model for only 155 epochs, this resulted in a 5.3% error rate on the validation set. 4.2
The validation errors displayed in Table 4 for the CNN runs (sf run 2,3,4,6) demonstrated a clear correlation between the number of epochs they were trained for and increasing performance (decreasing validation error). This is demonstrated visually in Figure 2. The test results for the six runs as supplied by ImageCLEF are displayed in Table 5. The CNNs demonstrated improved performance over the Softmax classi cation and their accuracy approximately corresponded to the amount of training that was performed. The test accuracy for both runs of the Softmax classi er were 37.56%. This indicates that despite cutting short the training for sf run 5 compared to sf run 1, both models had e ectively the same representation of the data when it came to classifying the test data. Compared to the validation results for the CNNs, the improvements with regard to the number of training epochs are not so clear-cut. For the model trained with learning rate of 0.005% there is a clear improvement between the test submitted at epoch 47 and the test submitted at epoch 55. However, the test accuracy decreased in the run submitted at epoch 59. 6 6.1
An examination of the validation error and test results in Tables 4 and 5 is very illuminating. Clearly the 70/30 training/validation method we applied was inappropriate in this case, as demonstrated by the signi cant variance between the validation and test performance. While it is possible that the test set was substantially di erent to the training set, it's more likely the 30% chosen for validation was not fully representative of the data. Given that the training data was not evenly distributed in all classes it is likely that the models over t the data corresponding to the more common classes and that the validation set was heavily skewed towards the common classes. However, we still believe that CNNs are suitable for this task despite the evidence of over tting in this case. Techniques for overcoming this issue are discussed in Section 7.
Having already pointed out the tremendous computational demands required by the CNNs, more robust validation procedures such as 10-fold cross-validation are clearly not feasible with the system employed in this test. That said, it may be possible to perform this kind of validation on a simpler model such as Softmax, in order to discover a more indicative training/validation split. Another option would be to take a more manual approach to splitting the sets, ensuring that all classes are evenly represented in the validation set.
It's worth noting that the models used for testing were only trained on the 70% training split. In future, we can expect better results by retraining the best model (based on some validation metric) on the entire dataset. 6.2
As alluded to out earlier, although the CNNs did not converge during training, they may have already begun to over t the training data with the result that the test performance actually decreased for the model at epoch 59 compared to the model at epoch 55. However, this is not entirely certain, as it is also possible that the model at epoch 59 was a better t for the validation data (table 4) at that point, but simultaneously a worse t for the test data. Had the models been able to train for longer, we may have had a clearer indication of their true performance. 6.3
The CNNs were able to extract improved representations from raw data without the requirement for domain knowledge. This is an important result both for this task and for MIR generally as it suggests that there is potential in using CNN or other deep learning strategies as a 'black box', whereby we will be able to achieve excellent machine learning performance without the need of expertdesigned feature extraction or domain knowledge.
We would have liked to train the network further, but need to prematurely halt the system for the purposes of submission. We believe that additional training would yield a better result. 7
We believe that that these results can be signi cantly improved upon by making use of a variety of techniques. Primarily we would want to explore training the CNNs using GPUs, as this will allow us to expand our hyperparamter and architecture search. Recti ed Linear Units (ReLUs), as opposed the Tanh units used in our network are also known to improve training performance [16, 17].
Although this network is very capable of learning quality representations
of the MNIST dataset, it is both less deep and less dense than networks used
to achieve state-of-the-art results in more sophisticated tasks [
Larger and deeper networks introduce issues with over tting, but we believe
this can be controlled using well-tried techniques such as dropout [
Finally, the validation method we utilised for these experiments did not produce an accurate understanding of the performance of our systems. In approaching this task in future we would be careful to construct a more representative validation set or use the 2015 test data for validation.
This work was supported in part by a Microsoft Azure for Research grant, which provided the cloud infrastructure to conduct our experiments. 12. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Over tting, J. Mach.
Learn. Res., vol. 15, no. 1, pp. 19291958, Jan. 2014. 13. Y. Bengio, Practical recommendations for gradient-based training of deep architectures, arXiv [cs.LG], 24-Jun-2012. 14. D. Ciresan, U. Meier, J. Masci, and J. Schmidhuber, A committee of neural networks for tra c sign classi cation, in Neural Networks (IJCNN), The 2011 International Joint Conference on, 2011, pp. 19181921. 15. Y. Lecun, L. Bottou, Y. Bengio, and P. Ha ner, Gradient-based learning applied to document recognition, Proc. IEEE, vol. 86, no. 11, pp. 22782324, Nov. 1998. 16. V. Nair and G. E. Hinton, Recti ed linear units improve restricted boltzmann machines, in Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010, pp. 807814. 17. A. L. Maas, A. Y. Hannun, and A. Y. Ng, Recti er Nonlinearities Improve Neural
Network Acoustic Models, W|&CP, vol. 28, 2013. 18. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, arXiv [cs.NE], 03-Jul-2012. 19. Classifying plankton with deep neural networks, Sander Dieleman. [Online]. Available: http://benanne.github.io/2015/03/17/plankton.html. [Accessed: 30-May2015]. 20. X. Glorot, A. Bordes, and Y. Bengio, Domain adaptation for large-scale sentiment classi cation: A deep learning approach, in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 513520. 21. Y. Bar, I. Diamant, L. Wolf, and H. Greenspan, Deep learning with non-medical training used for chest pathology identi cation, in SPIE Medical Imaging, 2015, p. 94140V94140V7.