This article describes the participation of the Computer Science Department of Morgan State University, Baltimore, Maryland, USA in the ImageCLEFcaption under ImageCLEF 2017. The purpose of this research and participation is to be able to predict the caption and detect UMLS concepts of an unknown query (test) image by using Cross Modal Retrieval and Clustering techniques. In our approach, for each image (without any caption or concept information) in the test set, we find the closest matching image in the training set by applying similarity search (e.g., content based image retrieval) in a combined feature space of color, texture, and edge-related visual features. By linking the associated caption and UMLS concepts of the closest matched image, further processing are performed to extract terms (keywords/concepts) to form a text feature vector and finally return the top ranked terms as predicted concepts (caption) from the best matching cluster centroids which are previously generated by applying K-means clustering in a term-document matrix of the training set. In this article we present main objectives of experiments, overview of these approaches, resources employed, and describe our submitted runs and results with conclusions and future directions.
This article describes the main objectives of cross modality matching approach
based on our first year participation in ImageCLEF 2017 [
Besides in clinical settings, bio-medical images are also sources of essential
information for research and education in biomedical literature. For example,
authors of journal articles frequently use images to elucidate the text and to
illustrate important concepts or to highlight special cases as Region of Interests
(ROIs) [
There has been a lot of interest in information retrieval, computer vision,
and multimedia community recently in developing cross and multi-modal
image retrieval techniques with the massive explosion of multimedia content on
the web. Multimedia contents, such as web pages, scientific publications, and
document images convey information using multiple modalities, including text,
layout/style and images. However, an intrinsic problem here is to investigate the
semantic correlation amongst the text and image data. Different models have
been proposed to learn the dependencies between the visual content of an image
set and the associated text captions, then allowing for the automatic creation of
semantic indexes for un-annotated images [
Motivated by these approaches in general domain, in the following sections, we describe our cross-modal search approach towards the concept detection and caption prediction tasks in ImageCLEFcaption for bio-medical images in journal articles. In the following sections, we describe our content-based visual search approach (Section 2) to link test images to closet match images in the training set, text feature extraction and K-means clustering in a term-document matrix (Section 3) from associated image concepts and captions and detection and prediction of concepts and captions respectively (Section 4) for test images based on using Python 3.5, with OpenCV 2.0, sklearn, scikit-image, NLTK and mahotas packages. Section 5 describes our submitted runs and results that we achieved and finally Section 6 provides our conclusions and future works. 2
We at first performed a content-based visual similarity search for each image in the test set as an example query image to a content-based image retrieval (CBIR) system where images in both training and validation sets are indexed at first by extracting several low-level color, texture, and edge related visual feature. The purpose of this similarity search is to find the closet matching image in the training (validation) set for each query (test) image and using its associated caption (concepts) for further processing (keyword extraction, clustering) for caption prediction and concept detection. So, the CBIR search is the first step of the pipeline of our cross-modal process to link unknown test images with associated captions and UMLS concepts of known images with captions (concepts). To save computational time, each image is resized (100 x 100 pixels) and the following features are extracted:
Local Color Descriptor: Instead of computing a color histogram for the entire image, a 3D normalized HSV color histogram is computed for different regions (1) the top-left corner, (2) the top-right corner, (3) the bottom-right corner, (4) the bottom-left corner, and finally (5) the center of the image as shown in Fig. 1. Each region is represented by a histogram with 8x × 12 × 3 = 288 entries where Hue, Saturation and Intensity values are quantized to 8, 12, and 3 bins respectively. Given 5 regions, the overall feature vector is 5 × 288 = 1440 dimensionality. Using regions-based histograms rather than global-histograms allows us to simulate locality in a color distribution.
In addition to the color descriptor, we extracted the well-known Local Binary
Patterns (LBPs) [
HOG is known as a keypoint descriptor in literature which expresses the
local statistics of the gradient orientations around a keypoint [
For similarity matching, each feature is concatenated to form a combined
feature vector and Euclidean distance measure is used for k-nearest neighbor
image similarity where top matching images are ranked from a low to high scores
in the range of [
(1)
Our next step of the process is the text feature extraction and indexing (creating
a document-term matrix for subsequent clustering) of associated image captions
(concepts) of training (validation) images and perform clustering to form
natural groups of images with similar (related) captions and concepts. For the
caption prediction task, each associated caption of training images is converted to
lower-case, all punctuation are removed and tokenized into its individual words.
Stopwords are removed using NLTK’s “english” stopword list and subsequently
terms are removed from the vocabulary that occur in fewer than 10 captions, and
finally the remaining words are reduced to their stems using NLTK’s Snowball
stemmer, which finally form the vocabulary list T = {t1, t2, · · · , tN } of index
terms or keywords of the image captions. Finally, a document-term matrix M
is created based on T , where each caption is modeled as a vector of keywords
(terms) as
where each wˆij denotes the tf-idf weight of a keyword ti, 1 ≤ i ≤ N in the
caption of image Ij . This weighting scheme amplifies the influence of terms,
which occur often in a document (e.g., tf factor), but relative rarely in the
whole collection of documents (e.g., idf factor[
T However, with a size of T > 20, 000, our matrix is still too large. Hence, to reduce the dimension further, Latent Semantic Analysis (LSA) is applied and further analysis (clustering) is performed in the LSA projected feature space by keeping explained variance above 90%. In many instances, LSA is found suitable to reduce dimensionality in a spare matrix and discover latent patterns in the data.
For the concept detection task, the text feature extraction and indexing approaches is more straightforward as we do not to perform any extra preprocessing steps, such as removal of stopwords, tokenization, stemming, LSA etc. In this case, we generate the vocabulary list T based on the presence of all the UMLS concepts in the images of the training set and generate the sparse matrix accordingly for further analysis in the next step of the process.
Our final goal is is to partition N data points (text feature vectors of training
set) into K clusters by applying K-means clustering [
After performing all the above processes (e.g., visual feature extraction, CBIR search, text feature extraction, and clustering), our final task is to detect concepts and predict captions of unknown images (without associated captions and UMLS concepts) in the test set. For this step, we at first find the closest matching image in the training (validation) set for each test image by applying the similar visual feature extraction and similarity search process described in Section 2. After finding the associated captions (concepts) of closet matching images, we next generate the text feature vectors accordingly and find the closet cluster labels by applying a minimum distance matching function to previously generated “centroid” or “center” in the training set as described in the previous section. Based on the cluster labels, we look-up for the top terms (keywords) and return those as assumed image captions or concepts. Below, all the steps of the above process is described algorithmically: Algorithm 1 Concept Detection/Caption Prediction Process 1: Initially, resize and extract color, texture, and edge-related visual features F (local color descriptor, HOG, and LBP) for images in the training set. 2: Extract text feature from captions (concepts) in the training set to generate the vocabulary list T and document-term matrix M 3: Apply K-mean clustering in matrix M to generate different number of clusters (nclusters). 4: for j ∈ S images in the Test Set do 5: Resize and extract visual features for test image Ij. 6: Extract text feature vector from associated caption (concepts) Dj by using vocabulary list T . 7: Find the closet cluster label by applying a minimum distance matching function between feature vectors of caption (concept) and cluster centroids. 8: Return and print the top (K) terms (keywords) from the best matching centroids. 9: end for 10: Finally, generate the result file (run) for test images with image name and associated caption (concepts). 5
This section provides descriptions of our submitted runs and analysis of the result. We performed, feature (visual and textual) extraction and clustering in both the training set of around 164K images and validation set of 10K images with associated captions and UMLS concepts. We submitted the following four runs for the concept detection task:
1. DET Morgan result concept from CBIR.csv : This is our baseline run for the concept detection task. In this run, each test image is automatically acted as a query image to our CBIR system to find the closet matching image in the training set and use the associated concepts as the concepts for the test image.
2. DET Morgan result concept from train Kmean top20.csv : In this run, each test image is automatically acted as a query image to our CBIR system to find the closet matching image in the training set and use the associated concepts to form a text feature vector. This vector matches to the closet cluster centroids out of 50 clusters previously generated by K-means in the training set and returns the top (20) terms (concepts) of that particular centroids.
3. DET Morgan result concept from val Kmean50 top15.csv : In this run, each test image is automatically acted as a query image to our CBIR system to find the closet matching image in the validation set and use the associated concepts to form a text feature vector. This vector matches to the closet cluster centroids out of 50 clusters previously generated by K-means in the training set and returns the top (15) terms (concepts) of that particular centroids.
4. DET Morgan result concept from train Kmean300 top15.csv : In this run, each test image is automatically acted as a query image to our CBIR system to find the closet matching image in the training set and use the associated concepts to form a text feature vector. This vector matches to the closet cluster centroids out of 300 clusters previously generated by K-means in the training set and returns the top (15) terms (concepts) of that particular centroids.
For concept detection task, evaluation is conducted in terms of average
(mean) F1 scores between system predicted and ground truth concepts in the
test set [
Our last run in the Table 1 ranked fourth group wise with a mean F1 score, 0.0498 when no external resources were used.
For the Caption Prediction task, we tried to submit few run based on following the same process described for the Concept Detection task. However, there were some problems in our runs (result files) and we received errors while the files were parsed by the evaluation tool as provided by the CLEF organizer. We are currently trying to fix the problem and will evaluate and analyze our results at a later time. 6
This article describes the cross-modal strategies of the CS Morgan group for the concept detection and caption prediction tasks of the ImageCLEFcaption track. Instead of directly performing image understanding, our cross-modal approach tries to link test images with images in the training set based on visual similarity at first and from there further text processing and clustering are performed to detect concepts and predict captions from groups(clusters) where images with similar concepts/captions reside. Our results indicate that clustering with large number (300) of centroids is better in terms of mean F1 score. However, contentbased approaches to image retrieval are not yet advanced enough to achieve the precision of text-based approaches, and we are currently working towards directly mapping image region to concepts aided by a ground-truth training set of image patches.
This research is supported by the NSF HBCU-UP Research Initiation Award
No. 1601044. We would like to thank the ImageCLEF [