In this paper, we describe our participation in the ImageCLEF 2015 Scalable Concept Image Annotation task. In this participation, we propose an approach of image annotation by using ontology at several steps of supervised learning with noisy unlabeled data. In this regard, we construct tree-like ontology for each annotating concept of images using WordNet and Wikipedia. The constructed ontologies are exploited throughout the proposed framework including several phases of training and testing of one-vs-all SVM classi ers. Several classi ers are trained on local or global visual features separately and results are ensemble using the classi ers' probability scores. The result turns out that our system achieves an average performance in this task.
Due to the explosive growth of digital technologies, collections of images are increasing tremendously in every moment. The ever growing size of the image collections has evolved the necessity of image retrieval (IR) systems; however, the task of IR from a large volume of images is formidable since binary stream data is often hard to decode, and we have very limited semantic contextual information about the image content.
To enable the user for searching images using semantic meaning,
automatically annotating images with some concepts or keywords using machine learning
is a popular technique. During last two decades, there are a large number of
researches being lunched using state-of-the-art machine learning techniques [1{
4] (e.g. SVMs, Logistic Regression). In such efforts, most often each image is
assumed to have only one class label. However, this is not necessarily true for
real world applications, as an image might be associated with multiple semantic
tags. Therefore, it is a practical and important problem to accurately assign
multiple labels to one image. To alleviate above problem i.e. to annotate each
image with multiple labels, a number of research have been carried out; among
them adopting probabilistic tools such as the Bayesian methods is popular [5{7].
More review can be found in [
Fortunately, some initiatives have been taken to reduce the reliability on manually labeled image data [10{13] by using cheaply gathered web data. Although the \Semantic gaps" between low-level visual features and high-level semantics still remain and accuracy is not improved remarkably.
In order to reduce the dependencies of human-labeled image data,
ImageCLEF [
In this paper, we describe the participation of KDEVIR at ImageCLEF 2015
Scalable Image Annotation, Localization, and Sentence Generation task [
The rest of the paper is organized as follows: Section 2 describes the proposed framework. Section 3 describes our submitted runs to this task as well as comparison results with other participants' runs. Finally, concluded remarks and some future directions of our work are described in Section 4. 2
In this section, we describe our method for annotating images with a list of semantic concepts. We divide our method into four steps: 1) Constructing Ontology, 2) Pre-processing of Training Data, 3) Training Classi er, and 4) Predicting Annotations. An overview of our proposed framework is depicted in Fig. 1. 2.1
Constructing Ontology Ontologies are the structural frameworks for organizing information about the world or some part of it. In computer science and information science, ontology is de ned as an explicit, formal speci cation of a shared conceptualization [16, 17] and it formally represents knowledge as a set of concepts within a domain, and the relationships between those concepts. To utilize these relationships in image annotation, we construct ontology for each concept of a prede ned list of concept used to annotate images.
In real world, an image might contain multiple objects (aka concepts) in a single frame, where concepts are inter-related and maintain a natural way of being co-appearance. We use these hypotheses to construct ontologies for Given Training Data: Large Scale Web Image Corpus Visual Features {f1, f2 ,.., fF}
Metadata Pre-processing of Training Data
Constructed Ontology Ontology of c1 Ontology of c2
Ontology of cN Processed Training Data
Visual Feature-wise classifiers training Training Data for c1
Training Data for c2
Training Data for cN
Classifiers for Feature f1
Classifiers for Feature f2
Classifiers for
Feature fF
Feature-wise trained models Models for Feature f1
Models for Feature f2
Models for
Feature fF !"#$%&'()"*#%+,#-(.%
Features
Annotation Prediction and
Ensemble
Final Annotations concepts [18]. In this regard, we utilize WordNet [19] and Wikipedia as primary sources of knowledge.
Let C be a set of concepts. We construct a tree-like ontology [20] for each concept cc 2 C. In order to build ontologies, rst of all, we select some types of relations including: 1) taxonomical Rt, 2) functional Rf , and 3) weak hierarchical, Rwh. The relations are extracted empirically according to our observations on WordNet and Wikipedia articles. For each type of relations, we extract a set of relationship property as listed below: Rt= f\inHypernymPathOf", \subClassOf", \isA" g Rf =f\habitat", \inhabit", \liveIn", \foundOn", \foundIn", \locateAt", \nativeTo", \liveOn", \feedOn"g Rwh=f\kindOf", \typeOf", \representationOf", \methodOf", \appearedAt", \appearedIn", \ableToProduce"g
Finally, we apply some \if-then" type inference rules to add an edge from a
parent-concept to a child-concept by leveraging the above relations.
Given a list of concepts, we select the potential images for each concept from
the noisy training images by exploiting their metadata (details about metadata
are given in [
We take the inverted index of image-wise weighted concepts, thus generate the concept-wise weighted images. To aggregate the images for a concept from three sources, we normalize the weight of images using Max-Min normalization technique, and linearly combine the BM25, MRR, and constant # to generate the nal weight of images. From the resultant aggregated list of images, top-m images are primarily selected for each concept.
Finally, in order to increase the recall, we merge the primarily selected training images of each concept with its parent concepts of highest semantic con dent (i.e. parents connected by rt 2 Rt) by leveraging our concept ontologies. Thus, we enhance training images per-concept as well as number of annotated concepts per-image. 2.3
Training Classi er Image annotation is a multi-class multi-label classi cation problem; current state-of-the-art classi ers are not able to solve this problem in their usual format. Towards this problem, we propose a technique of using ontologies during different phases of learning a classi er. In this regard, we choose Support Vector Machines (SVMs) as a classi er for its robustness of generalization. We subdivide the whole problem into several sub-problems according to the number of concepts, i.e. train SVMs for each concept separately, since using a large dataset at a time is not rational in terms of memory and time.
Another problem is that, along with the different parameters, the classi cation accuracy of SVMs depends on the positive and negative examples which are used to train the classi er. It is obvious that if classi ers are trained with wrong examples, the prediction will be wrong. However, selecting appropriate training example is formidable without any semantic clues. In this regard, for a concept, we take positive examples from its image-list which is generated in the preprocessing stage and the negative examples from all other concepts' imagelists those are not semantically related to the current concept. To handle this issue, we use our pre-constructed concept ontologies.
For each local or global visual feature, we train one-vs-all SVM for all concepts. With positive and negative examples, we train jF j probabilistic one-vs-all SVM models for each concept, where F is a set of visual feature types including CNN, GIST, Color Histograms, SIFT, C-SIFT, RGB-SIFT, and OPPONENTSIFT. We use LIBSVM [22] to learn the SVM models. As kernel, instead of using the default choice of Linear kernel or Gaussian kernel, since image classi cation 1 http://www.mpi-inf.mpg.de/yago-naga/javatools/index.html is a nonlinear problem and distribution of image data is unknown, we choose histogram intersection kernel (HIK) [23]. The HIK is de ned as: l kHI (h(a); h(b)) = ∑ min(h(qa); h(qb)) q=1 (1) where h(a) and h(b) are two normalized histograms of l bins; in context of image data, two feature vectors of l dimensions. 2.4
Predicting Annotations
The trained models for all concepts generated based on each visual features in
the previous subsection are used to predict annotations. Given a test image, if a
model of particular concept responds positively, the image is considered as voted
by current model i.e. the corresponding concept is primarily selected for
annotation. At the same time, the tracks of predicted probability and vote are kept.
This process is repeated for all learned models for all concepts. The concept-wise
predicted probabilities and votes are accumulated for all visual features. In
second level selection, empirical thresholds for accumulated probabilities and votes
are used to select more relevant annotations. Finally, we take top-k weighted
concepts as annotation for the test image. In ImageCLEF 2015 [
KDEVIR Runs and Comparative Results We submitted total ve runs, which are differ from each other in terms of: use of ontology or not; number of primarily selected training images, m; and based on textual, visual features or both; number of topK concepts selected for annotations. The con gurations of all runs are given in Table 1, where runs are arranged according to their original name to ease the ow of description. Here, run 1, 2, and 3 are employed based on both textual and visual features. However, run 4 and 5 are constructed based on the textual features of trained data, because both train and test dataset are same.
In Table. 2, evaluation results of our submitted runs are illustrated. It reveals
that \run 4" produces the best performance in terms of mean average precision
(MAP), although we did not use any visual features in this run. It shows the
effectiveness of our preprocessing stage of training data. However, the
performance of \run 1", \run 2", and \run 3" are not satisfactory. It turns out that
feature-wise learning of several visual features is not effective, although we could
not afford to process all visual features including CNN and SIFT variants due to
time constrain. Either, we need more effective visual features or more efficient
kernel and ensemble methods to boost up the performance of image annotation.
Details about all the performance measures are given in [
The evaluation result reveals that our proposed approach achieved an average performance among all submitted runs in terms of MAP 0.5 and MAP 0 overlap measures. However, in some runs, our system performance is not satisfactory. We could not afford to process all visual features due to time constraint. In future, we will consider deep learning to detect concepts in the noisy web images.
This research was partially supported by the HORI FOUNDATION of JAPAN, Grant-in-Aid C114. 16. Gruber, T.R.: Toward principles for the design of ontologies used for knowledge sharing? International journal of human-computer studies 43(5) (1995) 907{928 17. Studer, R., Benjamins, V.R., Fensel, D.: Knowledge engineering: principles and methods. Data & knowledge engineering 25(1) (1998) 161{197 18. Reshma, I.A., Ullah, M.Z., Aono, M.: Kdevir at imageclef 2014 scalable concept image annotation task: Ontology based automatic image annotation 19. Miller, G.A.: Wordnet: a lexical database for english. Communications of the ACM 38(11) (1995) 39{41 20. Wei, W., Gulla, J.A.: Sentiment learning on product reviews via sentiment ontology tree. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (2010) 404{ 413 21. Robertson, S.E., Walker, S., Beaulieu, M., Willett, P.: Okapi at trec-7: automatic ad hoc, ltering, vlc and interactive track. Nist Special Publication SP (1999) 253{264 22. Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM
Transactions on Intelligent Systems and Technology (TIST) 2(3) (2011) 27 23. Swain, M.J., Ballard, D.H.: Color indexing. International journal of computer vision 7(1) (1991) 11{32