Automating the detection of concepts from medical images still remains a challenging task, which requires further research and exploration. Since the manual annotation of medical images poses a cumbersome and error prone task, the development of concept detection system would reduce the burdens of annotation, interpretation of medical images while providing a decision support system for medical practitioners. This paper describes the participation of the CS department at Morgan State University, Baltimore, USA (Morgan CS) in the medical Concept Detection task of the ImageCLEF2020 challenge. The task involves generating appropriate Uni ed Medical Language System (UMLS) Concept Unique Identi ers (CUIs) for corresponding radiology images. We approached the concept detection task as a multilabel classi cation problem by training a classi er on several deep features extracted from using pre- trained Convolutional Neural Networks (CNNs) and also by training a deep Autoencoder. We also explored a Recurrent Concept Sequence generator based on using a multimodal technique of combining text and image features for recurrent sequence prediction. Training and evaluation were performed on the dataset (training, validation, and test sets) provided by the CLEF organizer and we achieved our best F1 scores as 0.167 by using DenseNet based deep feature.
Diagnostic analysis of medical images such as radiography or biopsy mostly
involve interpretations based on observed visual characteristics. In essence,
visual characteristics or features from images can be mapped to its corresponding
semantic annotations. Neural networks over the last two decades have been
successfully modeled to learn such mappings from data [
The dataset contains 64,753 radiology images from different modality classes as
the training set, 15,970 radiology images as the validation and 3,534 radiology
images from the same modality classes as the test set [
We approached the concept detection task by comparing elementary CUI multilabel classi cation and a recurrent CUI sequence generation using extracted features from varying deep learning architectures. The multilabel classi cation involves feeding the outputs from a feature extraction network into a fully connected network to obtain a sigmoid activation output representing the CUI label predictions. 2.1
Feature extraction is a critical component of medical image analysis. The
descriptiveness and discriminative power of features extracted from medical images are
critical to achieve good classi cation and retrieval performances. Instead of using
any hand-crafted features, transfer learning techniques can be used to extract
features of images from a relatively small dataset using pre-trained Convolutional
Neural Network (CNN) models [
Visual Feature Extraction To perform deep feature extraction, we chose
Densenet169 [
Vectors u and v are obtained by computing the weight vectors [a1; a2 an]
and [b1; b2 bn]. We selected the rst component 4096-D feature vector from
the PLS-CCA computation. The result obtained is representative of the features
from the maximized correlation of the DenseNet169 and ResNet50 features.
Feature Extraction based on Autoencoder We also use an
encoder-decoderbased framework (Fig. 1) to extract deep feature representations unique to the
dataset. Autoencoders are a type of unsupervised neural network (i.e., no class
labels or labeled data) that consist of an encoder and a decoder model [
The encoder region contracts normalized pixel-wise data from input images into smaller dimensional feature maps using sequential layers of 2D convolutions, batch normalization and ReLU activation. The output from the convolutional blocks is passed to a fully connected layer that represents a 256-D feature space. The decoder expands the 256 fully connected output by applying transposed convolutions that up sample the features back to the original input size. Batch normalization and ReLU activation are also added at each step of the transposed convolution sequence and the encoder lter sizes mirror the decoder lter sizes. The 256-feature output from the encoder is given as the auto-encoded deep feature representation of our input image. The Autoencoder was trained using the Adam optimizer [18] and a mean squared error loss on the ROCO training dataset for 20 epochs with a batch size of 50. The initial Adam learning was also set to 0.001.
Text Feature Extraction The deep text features are extracted from the image
concepts by learning and mapping deep feature embeddings that represent the
sequence of image concepts. The embeddings are learned during training when
a xed length of CUI sequence is passed to a neural embedding layer. Before
passing the CUI sequences to the embedding layer, for each input image, the
image concept sequence is tokenized using the Keras text preprocessing library.
Since a xed length of tokenized CUI sequence was required for the embedding
layer the differences in CUI sequence length for different input images was
accommodated by zero-padding the tokenized sequence up to the maximum CUI
sequence length of 140. During training, the embedding layer uses a mask to
ignore the padded values and its output is passed to a long short-term memory
(LSTM) layer [
The fully connected classi er was trained over 20 epochs with a learning rate of 1e 3 for an Adam optimizer. Since the concept set was split into groups and a different classi er trained for each concept group the overall CUI prediction for an input image involves the combination of the predictions from all concept group classi ers. 2.3
The CUI sequence generator involves training a recurrent classi er on a fusion of the extracted image features and the embedded textual features. The text features are obtained by learning the embeddings at training time from the embedding layer stacked with a LSTM layer to give a 256-D text feature output. Since concatenating the image and text feature vectors would require equal feature vector lengths, to combine the 256-D text features and the 4096-D image features, the 4096-D image feature is down-sampled to 256 by passing it through a dense layer with 256 units to give a 256-D feature output. The 256-D image feature and 256-D text feature is passed to a concatenation layer to obtain a 256D output that is passed to a nal dense classi cation layer for the prediction of the next word in the CUI sequence. The CUI sequence prediction begins when a start signal is passed as the rst element in the CUI sequence and the prediction ends when a stop signal is predicted by the classi cation model as shown in Fig. 3. The recurrent classi er was trained over 30 epochs with a learning rate of 1e 3 for an Adam optimizer and a batch size of 50. 3
Using the provided test dataset, multiple runs were submitted based on the multi-label classi cation with DenseNet, DenseNet-ResNet and Auto Encoded features. The result from the recurrent concept sequence generator with DenseNet encoded features was also submitted and the F1 evaluations are represented in Table 2. Our best result with a F1 score of 0.167 was obtained from the multilabel classi cation of DenseNet feature.
Run Method F1 Score MSU dense fcn Densenet169 + multilabel classi cation 0.167 MSU dense resnet fcn 1 (Densenet169 + Resnet50) + multilabel classi cation 0.153 MSU dense feat Densenet169 + multilabel classi cation 0.139 MSU dense fcn 2 Densenet169 + multilabel classi cation 0.094 MSU dense fcn 3 Densenet169 + multilabel classi cation 0.089 MSU autoenc fcn Autoencoder + multilabel classi cation 0.063 MSU lstm dense fcn Desnet169 + Recurrent concept generator 0.062 1. MSU dense fcn: This run utilized a multi-label classi cation model with the training parameters (described in section 2.2) based on the features extracted from a pre-trained DenseNet169. The threshold for the prediction score is set at 0.4 for the multi-label sigmoid classi cation which ranged from 0 to 1. Concept labels with prediction scores less than 0.4 are considered irrelevant to the input image. 2. MSU dense resnet fcn 1: In this run, the PLS-CCA of DenseNet169 and ResNet50 features ar computed to obtain fused features for the multi-label classi cation. The prediction score threshold for this run is also set at 0.4 for the nal multi-label sigmoid classi cation 3. MSU dense feat, MSU dense fcn 2, MSU dense fcn 3: These runs are variations of the MSU dense fcn run with different prediction score thresholds of 0.5, 0.3 and 0.25 respectively. 4. MSU autoenc fcn: The encoder-decoder model is utilized for this run to obtain the encoded features of the input images. The multi-label classi cation model (with parameters same as in runs 1,2 and 3) is trained on the Autoencoded features. The threshold for the prediction score from the classi cation model is also set to 0.4. 5. MSU lstm dense fcn: This run involved the recurrent generation of concepts by utilizing image features extracted from DenseNet169 combined with embedded concept sequences as described in 2.3. The obtained results clearly show the concept prediction challenge as more of a classi cation problem than a sequence generation task since all multi-label classi cation approaches performed better. 4
This article describes the strategies of the participation of the Morgan CS group for the concept detection tasks of ImageCLEF2020. We performed multi-label classi cation of CUIs in different deep feature spaces. We achieved comparable results considering the limited resources (computing and memory power) we had at the time of the submission. Since the ROCO data set is grouped into different modalities, we plan to perform separate multi-label classi cation for the different modalities in future.
This work is supported by an NSF grant (Award ID 1601044), HBCU-UP Research Initiation Award (RIA). 15. R.H.R. Hahnloser, R. Sarpeshkar, M.A. Mahowald, R.J. Douglas and H.S.
Seung: Digital selection and analogue ampli cation coexist in a cortexinspired silicon circuit. Nature. 405, pp. 947{951 (2000). https://doi.org/doi: https://doi.org/10.1038/35016072 16. F. Chollet: keras, GitHub. https://github.com/fchollet/keras. Last accessed 29 Jul 2020 17. H. Hotelling : Relations Between Two Sets of Variates. in Breakthroughs in Statistics: Methodology and Distribution, S. Kotz and N. L. Johnson, Eds. New York, NY: Springer, pp. 162|190 (1992) 18. D.P. Kingma and L.J. Ba : Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG] International Conference on Learning Representations (ICLR), (2015)