In this paper, we present the LIP6 annotation models for the ImageCLEFannotation 2010 task. We study two methods to train and merge the results of different classifiers in order to improve annotation. In particular, we propose a multiview learning model based on a RankingSVM. We also consider the use of the tags matching the visual concept names to improve the scores predicted by the models. The experiments show the difficulty of merging several classifiers and also the interest to have a robust model able to merge relevant information. Our method using tags always improves the results.
Last year, in ImageCLEFannotation 2009, we focused on how to deal with
imbalanced data [
In this work, we consider two models that merge the predictions of several
classifiers. The first model takes the mean of the predicted score of the classifier,
where each classifier is trained on a specific visual feature space using the labeled
data provided for the competition. The second model makes use of additional
unlabeled data to train the classifier in the semi-supervised, multiview paradigm
[
In addition to visual descriptors, the text associated to an image is often
relevant to improve image retrieval. There are only a few works [
The remainder of the paper is organized as follows. In Section 2, we propose our models for image annotation using multiple views. Section 3 describes our method which uses tags to improve annotation. The experiences are illustrated in Section 4. The conclusion and perspectives are presented in Section 5.
The data for image annotation is often highly imbalanced: for many classes,
there are only very few positive (or negative) examples. As we showed in [
Strictly speaking, a Ranking SVM does not learn a classifier. A classifier can however be obtained by comparing the scores with an appropriate threshold. In the following, the classifier is obtained by comparing the score to 0: if an observation x is such that hw, φ(x)i > 0, then we predict that x is in the positive class, otherwise we predict that it is in the negative class. Although this choice may not be optimal, it is a simple decision rule that gives good results in practice.
Last year [
To obtain a baseline, we perform several RankingSVM on different visual features, then we create a fusion of the outputs using an arithmetic mean. Figure 1 describes the fusion model we use in the ImageCLEFannotation2010 task. 2.2
Images can be described in several visual feature spaces (like SIFT, HSV, ...). Each of these representations, or views of an image, is very informative about the label of the image, but the different views provide rather independent information. Semi-supervised multiview learning aims at using these characteristics of the different views in order to improve the accuracy of the classifiers. The main principle is as follows: after training classifiers on each view independently (using standard supervised learning, in our case RankingSVMs), these classifiers are modified so that they predict, as much as possible, the same class labels on the unlabeled data. In our multiview learning model, we get the different views
The algorithm we use for the semi-supervised procedure is an iterative procedure. After the initial training step of the different classifiers on each view, each iteration of the algorithm consists of (1) predicting on all available unlabeled examples, then (2) adding to the training set (and removing from the unlabeled set) all examples for which all classifiers agree on a given class label. Equivalently, the unlabeled examples are given predicted labels based on the unanimous vote of the different classifiers. After step (2), the classifiers are re-trained on the new training set, and steps (1) and (2) are repeated. The algorithm stops when there are no more unlabeled examples on which all the classifiers agree. The process is described in Figure 2.
The rationale of this procedure is that when all the classifiers agree on a class label, the label is likely to be correct. Then, adding those examples to the training set gives us more training examples. Moreover, because we systematically add to the training set new examples on which the classifiers already agree, the procedure tends to increase the agreement between the different classifiers on the original unlabeled dataset.
In practice, the semi-supervised learning procedure improves performance when the classifiers are sufficiently accurate so that most training examples added at each iteration are correctly classified. If this is not the case, multiview learning may actually decrease performance, since we train new classifiers on more noisy training sets.
In the ImageCLEFannotation 2010 task, the observations are composed of the multiple views of image: each view can be considered as a feature type. We perform RankingSVM on each feature type using the training dataset. The training dataset is the labeled dataset and the test dataset is the unlabeled dataset. 3
To improve image annotation, the image associated text can be used. Free texts, tags and visual concepts are three kinds of text we can use for image annotation, but there is a big difference between them. 1. Free text is based on a very large vocabulary, it can be composed of a description of the image - this type of free text is often relevant to improve the image classification - or it can be composed of the associated text (like web pages) - this type of free text is often fewly relevant. 2. Tags are a little bit more specific then free text. When an image is associated with a tag, this tag is most of the time relevant for this image, but when a tag is not associated with an image, it should not mean that this tag is not relevant for the image. 3. Finally, visual concepts come from a very specific vocabulary, and contrary to tags, when a visual concept is not associated with an image, it means that this visual concept is not relevant for this image.
We then deduce that if the name of a visual concept tagged an image, then
this image should be annotated with this concept, but the contrary is false. Our
method is based on this idea. If a tag maches the name of a visual concept,
then the images associated with this tag will have their prediction scores for this
concept set to a given value we called U P . In order to improve the matching
between the name of the visual concepts and tags, we first apply - on both
concept names and tags - standard text processing, such as porter stemming
algorithm [
The corpus is composed of a training set of 8000 Flickr images and a test set of 10000 Flickr images. We split the training set in a training set of 5000 images and a validation set of 3000 images1. Each image is annotated in average by 12 visual concepts chosen among the 93 hierarchical visual concepts. 4.1
We extract three different types of visual features from each image. 1 The images in the validation set are the same as ImageCLEFannotation 2009 HSV First, we segment images into 3 horizontal regions and extract HSV features. For each region, we compute a color histogram in the HSV space. We believe that these visual descriptors are particularly interesting for general concepts (i.e. not objects), such as: sky, sunny, vegetation, sea and etc. SIFT Second, we extract SIFT keypoints, and then we cluster them to obtain a visual dictionary: we extract the SIFT keypoints of each image. To reduce the size of the dictionary and avoid duplicate keypoints, the keypoints are clustered with a nearest-neighbors algorithm, to obtain 1024 clusters. Then, each cluster represents a visual word in the dictionary, and each image is indexed using this dictionary.
Mixed+PCA Third, we use a concatenation of various visual features from 3
labs proposed by the AVEIR consortium [
We first apply - on both concept names and tags - standard text processing,
principally stemming using Porter stemming algorithm [
Another information is the number of predicted scores modified by the tags. For example, in the validation set, there are 3000 images and 93 concepts, so there are 279000 predicted values. Among those, only 1829 are modified when we use stemming. On average over the different concepts, 19.7 predicted scores are modified. Even though the tags affect rather few values perc concept, we will see in the next section that those modifications are most often relevant.
Figure 3 compares the number of images which contain a tag corresponding to a given concept (for example, if we consider the concept “Sunny”, it is the number of images which are tagged with “Sunny”), and the number of images relevant for this concept in the training set (for example, if we consider the concept “Sunny”, it is the number of images which are labelled are positive for the concept “Sunny” in the ground truth of the training set). This figure shows that, contrarily to the intuition, the number of images tagged with a given concept is not correlated to the number of relevant images for this concept. As we said in the previous sections, we use RankingSVM as the basic learner. We have trained RankingSVMs on all three types of features in a fully supervised setting (the fusion model) or in the semi-supervised setting (the multiview model). For both types of training, the different classifiers are merged in a single model by taking the mean of the predicted scores. These scores are finally normalized using a gamma distribution. We optionally use tags.
2000 3000 number of relevant images (training set) 4000 5000 that the multiview model is less accurate than the Fusion or Sift models. This may be due to the use of unanimous vote. 4.4
In ImageCLEFannotation 2010, we submitted the following 5 runs : Run1 We perform a RankingSVM using only SIFT features.
Run2 We perform a RankingSVM for each type of features (HSV, SIFT, Mixed + PCA). Then we merge the prediction scores using an arithmetic mean. Run3 We perform the multiview learning by using three views (HSV, SIFT, Mixed + PCA); each view represents a type of visual features.
Run4 Same as Run2, but here we also consider tags. The prediction score given by the classifier for a given concept is increase up to 1, if the image is tagged with the name of the visual concept. Run5 Same as Run3, but here we also consider tags. The prediction score given by the classifier for a given concept is increase up to 1, if the image is tagged with the name of the visual concept.
Table 3 gives the official results on test set. We note that the results are close to the results obtain by a random run. We can conclude that there might be some mistakes in our process (maybe in the final step where we have to make the prediction scores in a given format), but not in our methods because the results on validation set are reasonable (see Table 2). In Table 3, we can notice that the use of the tags improves significantly the results.
We proposed two models to train and merge the results of different classifiers in order to improve annotation. We described a multiview learning method for image annotation in which each view is a visual feature type. We also merged the predicted scores for all feature types. The experiments showed that our multiview model is close to, but less effective than, the fusion model. Maybe the voting part should be modified to obtain higher performances.
We also considered the use of the tags matching the visual concept names to improve the scores predicted by the models. Our method using tags always improved the results of the classifiers. Our study of the links between visual concept names and tags showed that when an image is tagged with a concept name this image should be labeled with this concept.
As perspectives, we can modify the voting part in the multiview model to avoid adding noisy examples to the training set. Moreover, we intend to try our models with other feature types, such as text, to study the evolution of the performances.
This work was partially supported by the French National Agency of Research (ANR-06-MDCA-002 AVEIR project).