This paper describes the participation of the MindLab research group of Universidad Nacional de Colombia at the ImageCLEF 2014 Scalable Concept Image Annotation challenge. Our strategy mainly relies in finding a good visual representation based on deep convolutional neural networks. Despite the simplicity of the proposed classifier which allows to deal with the large-scale nature of this task, we can achieve good performance (our proposed approach achieved the best MAP) thanks to the rich visual representation based on learned features.
This paper describes the participation of MindLab research group of
Universidad Nacional de Colombia in the 2014 version of the Scalable Concept Image
Annotation challenge at ImageCLEF [
In this work we proposed a transfer learning approach by using a convolutional network that was trained over a million of images of the ImageNet dataset to enrich the visual representation of the images from this particular domain.
The rest of the paper is organized as follows: Section 2 describes the characteristics of the dataset; Section 3 describes our multi-label annotation approach; Section 4 presents the experimental results; and finally, Section 6 presents some concluding remarks.
The dataset is composed by a subset of images extracted from a database of millions of images downloaded from the Internet. For each image, the corresponding web page that contained the image is available o↵ering a set of unstructured and noisy related text. This is a large training dataset composed by 500,000 images with meta-data but without labels. To evaluate the proposed systems, two sets with di↵erent list of images and corresponding concepts are provided: Development Set. This set is annotated and composed by 1,000 images labeled with 107 di↵erent concepts.
Test Set. Is an unlabeled set composed by 4,122 unique images and 207 possible concepts.
To validate the scalability of the proposed systems, the list of concepts are die↵rent for the development and test sets, moreover, within each set the list of concepts will not be the same for all images. 3
3.1
Although several sets of pre-processed visual features are provided by the challenge organizers, our strategy is based on building our own visual representation based on DCNN. And, for this strategy, we rely in the theory of transfer learning which is based in the ability of a system to recognize and apply knowledge learned in previous domains to novel domains, which share some commonality.
We use the Yangqing Jia et al. [
This convolutional network has 60 million parameters and has an architecture composed by eight layers: five convolutional layers and three fully-connected. The output of the last fully-connected layer is the input for a 1000-way soft-max layer which produces a distribution over the 1000 classes (normalized scores).
Each image is scaled so that the smallest dimension has 227 pixels preserving the original aspect ratio. This raw scaled image is given as input to the network. We used the last fully-connected layer activations, composed by 4096 neurons, as the visual representation for each image. 3.2
As text representation for images we processed the provided word-score features based on term frequency, DOM attributes and word distance to the image. This text is preprocessed using stop-words removal and stemming, generating a final list of words for each image that is used as textual annotation. Notice that this training set is noisy, that means that a lot of incorrect words could be associated to an image. 3.3
The process to assign labels to images is as follows: the list of query concepts are stemmed and compared to the list of words of the textual annotation obtained in 3.2, if the query concept is presented in the list of words assigned to an image, this concept is assigned as label to the corresponding image. After the label assignment process, if some image does not have any concept, then this image is removed from the original training set. This leads to a total of 383,815 filtered training images to train the annotation model for the development set composed by 107 concepts; and a total of 427,444 training images for the test set composed by 207 concepts. Once we extracted a training set composed by the filtered images which are represented by the visual features generated through the convolutional neural network and annotated with its corresponding concepts, we trained a logistic regression model with multiple outputs, which produces a distribution over the 207 di↵erent concepts of the test set. Later, visual features are extracted for the test images and the annotations are predicted by using the trained logistic regression. 3.5
The logistic regression layer produces a distribution with denotes the probability of belonging to each concept, in order to give a final decision in the annotation, it is necessary to define a threshold value. To define an appropriate value for this threshold we perform an exploration by using the development dataset. Figure 2 shows the results of the exploration, using three di↵erent strategies: 1) the output of the logistic regression is normalized by samples assigning 1 to the maximum value and 0 the minimum value 2a; 2) the output for each concept is normalized by setting 1 to the maximum value achieved among all samples (2b), and 3) the logistic regression output is used directly, without normalization (2c).
The die↵rent performance curves in figure2 gives clues to select the score normalization method. Normalizing either by concept or sample give better re(c) sults than no normalization at all. And among those two types of normalization, sample normalization yields a curve which is less sensitive to slight threshold variations, allowing more tolerance in the choice of the threshold. 4
We submitted 2 runs, where the only di↵erence is the strategy used for threshold assigning: Run 1 (MindLab 01): In this run we assigned the best threshold found in the exploration performed in the development set using per sample normalization. This threshold was used for all concepts.
Run 2 (MindLab 02): In this run the output for each concept is also normalized per sample as in the first run, but two types of thresholds are used. The best threshold found for the development concepts with sample normalization was used for the new concepts in the test set. For the concepts present in development and test, an specific threshold was used for each concept. This specific threshold was fine tuned for each development concept to achieve the best performance in the development images.
The ocial results of both submitted runs are reported in Table 1, also, the best result obtained among all the submissions is reported for comparison. As can be seen from the table, our strategy achieved a better result in MAP value than the best submission obtained among all the participants, but a more adequate strategy is required for selecting the final annotations. Figures 3 and 4 show the obtained results of precision and recall for both submissions grouping by concept or sample. When comparing performance of both submissions grouping by concept (figures 3a and 4a), an improvement in recall is evident for the second submission. This can be attributed to the specific threshold used for the concepts on the second submission. This improvement in recall is also present when comparing the performance of both submissions grouping by sample (figures 3b and 4b). But this comparison also reveals a drawback of the strategy used for the second submission, as precision drops significantly. This results bring forward the need to evaluate other strategies for threshold selection, which do not sue↵r this kind of disadvantages. 5
In this work, we proposed a method for multi-label annotation. Despite the simplicity of the proposed classifier which allows to deal with the large-scale nature of this task, we can achieve a good performance (our proposed approach achieved the best MAP) thanks to the richness of the visual representation based on learned features via deep convolutional neural networks.
The experimental results showed that a good performance can be achieved by applying knowledge from other similar domain (transfer learning).
Run
Position MF-samples (%) MF-concepts (%) MAP-samples (%) MindLab 01 MindLab 02 8 10 1 This work was partially funded by project Multimodal Image Retrieval to Support Medical Case-Based Scientific Literature Search, ID R1212LAC006 by Microsoft Research LACCIR and Jorge Vanegas and John Arevalo also thanks for doctoral grant supports Colciencias 617/2013. Sebastian Ot´alora also thanks Colciencias for its support through the grant “ J´ovenes Investigadores 2012” in call 566.