In this paper we report on MICC participation to the Scalable Concept Image Annotation subtask of the ImageCLEF Photo Annotation and Retrieval competition [13]. Our goal has been to investigate the applicability of data-driven methods that have obtained good results in the eld of social image annotation and retrieval to web images. These methods have been applied typically to tasks such as tag ranking, tag suggestion and re nement. Since they do not require a training stage they can be applied in cases in which the set of annotation keywords can vary greatly over time or when the set of images to be analysed is very large.
This paper describes our participation in the Scalable Concept Image Annotation
subtask of the 2013 ImageCLEF competition [
Recently, data-driven approaches have shown to be able to deal with very
large scale scenarios, and have been applied to tag ranking for social image
retrieval, tag re nement for social image annotation [
Our approach, described in section 3, computes a visual distance between test images and train images and then obtains a score for several words in WordNet by performing a simple density estimation. Afterwards, a nal score for each of the concepts from the requested vocabulary is obtained by evaluating several semantic similarities. Section 3 describes in more detail the various steps of the method; Section 4 reports the experimental setup used, while description of runs and results are reported in Section 5. Conclusions are drawn in Section 6.
Given a set of training images I with their respective web pages, a set of test images IT and a vocabulary of words V the goal is to get a relevance value r(i; w) 8w 2 V; 8i 2 IT and to choose a set of nal annotations to be assigned. The latter ones can be simply obtained by using a threshold on relevance values or by enumerating the rst xed N words, where N has been determined empirically. Our method is comprised of four steps: 1. Building a set of arti cial tags for every image in the provided training set.
This casts the problem as a tag re nement task. 2. Features extraction from training and test images and computation of an image { and corresponding web pages { neighborhood for every test image. 3. Construction of a candidate words set by image annotation, based on TagRelevance method over text features. 4. Filtering of stop-words and re-ranking of words by using several semantic metrics, de ned on WordNet and Wikipedia ontologies.
The obtained scores correspond to the nal relevance value assigned. Particular attention is given to the issue of scalability: our approach can scale up to utilize as many features and data as possible. Note that we are also assuming an openworld vocabulary which comprises potentially every possible word used on the web. However, as English WordNet and English Wikipedia are more mature, we consider only words which are contained in these two ontologies. This limitation can be possibly overcame by exploiting a system for automatic translation like Google Translate or Bing Translator; however, we have not used this approach in this work. 3
The basic idea of nearest-neighbor methods is to select a set of visually similar
images and then to select a set of relevant associated words based on a word
transfer procedure. Images selected as visually similar must be tagged with a
set of words possibly related to the content. This type of methods has also
been applied to di erent tasks such as tag suggestion and tag ranking/relevance,
applied to the context of social media [
In this ImageCLEF task, however, training images don't have any words (or tags) associated, instead they have one or more web pages with natural language text content. To overcame this issue, the rst step is to build a training set of arti cially labeled images to be used as a source of neighbors. Text, metadata and URLs from web pages are transformed in a set of tags for every image. It is not required to have perfect annotations as nearest neighbors method can make use of bigger training set, by simply using a bigger visual neighborhood sample to better estimate the speci c tag distribution. A source of noise is related to the kind of relationship between the images and corresponding web pages: some images can possibly be completely disassociated to the content described in the web page. As there's no easy way to determine if this is the case, some images can possibly be arti cially tagged with completely unrelated tags. In our experiments we directly employed a set of text features provided. 3.1
Li et al. have proposed a tag relevance measure for image retrieval based on the
consideration, originally proposed in [
Di erently from the original approach of [
The core of a working nearest neighbor approach is given by the space where
images are represented and by the selection of a good distance measure. In
our experiments we use only a visual space derived directly from the visual
features provided with the ImageCLEF datasets. An early fusion is made by
concatenating all the features provided (global color histogram, getlf, CSIFT,
GIST, opponent SIFT, RGB-SIFT, SIFT) resulting in a 21,312 dimension space.
All features are singularly normalized using `2 norm. In our implementation the
distance between images is computed using a Gaussian kernel:
d(Ii; Ik) =
ejjfi fkjj
2
(2)
where Ii is the visual neighbor in the i position, with N features fi = (fi1; : : : ; fiN ),
and is set as the median value of all the distances. The size of Development
and Test sets, together, is only of 3; 000 images and the training set is comprised
of 250; 000 images, which constitute a total of 3; 000 250; 000 = 750; 000; 000
distances. Given the relatively low number, we directly computed all distances
exhaustively. The process took about three hours on a medium spec computer.
In alternative, as the training set increases, one can use an approximate
technique like LSH[
After some initial experiments we have used the square of the distance also to weight the occurrences of a tag t in the neighborhood of an image Ik. 4.1
We used three kind of provided features: the score features, the triplets used to get images in search engines and the training URLs to create the set of tags associated with the images. The URLs were processed to extract the words that composed them by means of regular expressions and by checking their presence in WordNet. 4.2
To cope with the fact that the tag re nement approach used is applied to images that are associated with a set of textual features that could be di erent from the set of keywords to be used for annotation, we have tested some approaches to perform simple semantic augmentation of the tags resulting from the process described in Section 3.1.
Initially we have tried to add WordNet synonyms to the list of selected tags.
However this approach has resulted in very limited improvement. A second
approach has provided some steady improvement when using the Test dataset and
therefore has been used also in all the runs on the Development dataset: 10 tags
with the highest tag relevance score are selected, then the ImageCLEF keywords
that have an overall strong semantic similarity with them are added to the list of
candidate words used for annotation. This selection is performed by computing
the average semantic distance between all the tags, considering the best
semantic relatedness based on Wikipedia article internal links structure [
Nearest neighbor approaches have proven to be able to use several distances by
fusing the results of more than one system [
We submitted ve runs, using both Development and Test datasets. In all the runs we used all the pre-computed features to evaluate the visual neighborhood, score features were used as image tags and tag relevance was computed weighting the presence of tags using the squared visual distance between the image to be annotated and its visual neighborhood.
Only two parameters were varied during the runs: the number of tags assigned to each image and the size of the visual neighborhood. 1. Run 1: 2000 NNs, 5 tag per image. 2. Run 2: 3000 NNs, 10 tag per image. 3. Run 3: 3000 NNs, 7 tag per image. 4. Run 4: 4000 NNs, 7 tag per image. 5. Run 5: 4999 NNs, 7 tag per image.
Results in terms of F1 micro, F1 macro and MAP are reported for Development and Test datasets in Table 1 and 2, respectively. It can be observed that the larger the number of visual neighbors the better the performance. The improvement is much reduced on the Test dataset probably due to the over tting induced by using the same set of images from the Training dataset to compute the visual neighborhood in both experiments.
The system has been completely developed in Python, without attempting to implement any particular optimization. Running all the experiments on a portable PC with 2.53 GHz Intel Core i5 processor takes about 2.5 hours.
Run F1 micro F1 macro MAP In this paper we have presented our system for web images annotation based on a data-driven approach that has been used for tag reranking in the context of social media. Thanks to its simplicity and the fact that it requires no training or supervision, the system can be executed on mid level PCs and can be easily applied to other datasets. The system has also just two main parameters that have to be adjusted: the number of images used to create the visual neighborhood of the images to be annotated and the number of tags to be selected for annotation.