Visual annotation is still an open issue. The Content Based community admits that a plurality of features and systems shall be considered. We present in this paper four very different strategies using not only visual information but also text, to implement ImageCLEF2009 Photo Annotation Task. The visual features are various, such as HSV, Gabor, EDGE, SIFT, and some more recent. Then we study each model performances, and propose a new measure, the Carefulness Index (Q) computed on the histogram of the model's outputs. Q seems to be correlated with the model performances.
and propose a new measure, the Carefulness Index (Q) computed on the histogram of the model’s outputs. Q seems to be correlated with the model performances. We also analyse simple fusion models. In average the best model is the simplest, the arithmetic average, compared to the selection of the a priori best model, or an early fusion model.
The next section presents the four models, then the results are analysed and the Carefulness Index measure is proposed in section 4. Other comments on the models performances are given before to conclude. 2 2.1
We used the color-based visual descriptors in this model. As in [
We train a SVM classifier1 which has a linear kernel.Because of the imbalanced class problem, we use a ROC area as the loss function as proposed in [1]. So we consider not only the misclassification in each learning iteration, but also the number of positive and negative examples in order to avoid the fault ignorance. ROCarea can be computed from the number of swapped pairs
SwappedP aires = k{ (i, j) : (yi > yj)and(wT xi < wT xj )}k i.e. the number of pairs of examples that are in the wrong order
ROCarea = 1 −
SwappedP aires #pos.#neg . Here 1-ROCarea is used as the value of misclassification in loss function for each iteration. More details on this model can be found in [2]. 2.2
This model uses three sets of features: The first one is based on 512 bins RGB histogram of the three horizontal stripes (same height of 1/3 of the image height, whole width of the image as presented in previous section). Histograms are normalized and they result is a 1536 histogram. The second set of features are SIFT, using software provided by K. van de Sande [SAND08]. The SIFT features are extracted from regions selected according to Harris-Laplace feature points detection. Each feature is a 128-dimension vector. A visual vocabulary containing 4000 dimensions was then generated using the SIFT features of the learning set, yielding to a 4000 dimensions vector for each 1http://svmlight.joachims.org/svm_perf.html image. The third feature set, called HSVGAB, is an early fusion of colour and texture features. We used a 64 dimensions HSV colour histogram concatenated with a 40 dimensions vector describing gabor filters energy (7 dimensions, 5 scales). For the RGB, SIFT, and the HSVGAB features we used then a simple one against all SVM (RBF kernel) that learns the probability for one sample of belonging to each concept. For the SIFT features, we used additionally a multiple SVM leaning process. Consider one concept C having pc positive samples, and nc negative samples (ni = 5000 - pc ¿¿ pc). We define Nc SVM with all the positive samples and 2*pc negative samples, so that union the negative samples as all SVMs cover all the pc negatives samples of C. Each of these SVM learns the probability of belonging to each class concept/non concept. For one concept, we sum-up then the results for all the NC SVMs. We applied then a scaling in a way to fit the learning set a priori probabilities. Then we select the best feature/learning combination for each concept. We took into account the hierarchy of concept in the following way: a) when conflicts occur (for instance the tag Day and the tag Night are associated to one image of the test set), we keep unchanged the larger value tag, and we decrease (linearly) the value all the other conflicting tags, b) we propagated the concepts values in a bottom-up way if the values of the generic concept is increased, otherwise we do not update the values. More details can be found in [4]. 2.3
This model is an average of only visual information models, based on SVM and Visual Dictionary approaches on some new features depicted in [5]. As some of these models were proposed for the first time, we decided to build for this paper an average model that is the arithmetic average of three sub-models.
In sum, the Model 3 is built from various visual features: HSV, EDGE, Gabor, and the
recent DF and Profile Entropy Features (PEF) [
This model 4 is focused on global image descriptors and favors fast algorithms that can scale to thousands of images and annotation concepts. First, we represent each image using a text descriptor and a global visual descriptor. As visual descriptors we use global color, texture and shape features, similar to those presented in MPEG7. We use Canonical Correlation Analysis (CCA) to infer a latent space where the two representation are most correlated. Given the visual features of an unseen image, we fist project it to the CCA space and then we infer the linear combination of input concepts that is most correlated with it. We then back-project the result into the input space and we normalize it to [0, 1]. A value close to 1 means that the corresponding concept is likely to be found in the image, while a value close to zero suggests the contrary.
The tradeoff in our method is a slight loss of precision, but we make up for this in speed (we use less than 1 sec. for both training and prediction on an average 2.5 GHz PC). Moreover, adding new concepts to our method is straightforward and do not require training separate models for each concept. More details can be found in [6]. 3 3.1
We give the average Area Under the Curve (AUC) of the four models in figure 2 for each topics, and their average in table 1 including comparison to the best runs of each team participating to the campaign. We see that AUC(Model 1) > AUC(Model 2) > AUC(Model 3) > AUC(Model 4).
For comparison, three basic fusion models are computed. The first one, called early fusion, is a SVM trained on the merged features of the four models. The second is the simple arithmetic average of the outputs of the four models (late fusion). The last one, called ’best1’ is the selection of the best model according to the training performances.
The best fusion of the four models is the late fusion which gives an average AUC of 0.55, and occupies the 12th rank among the 19 teams in the official VCDT evaluation. Anyway, it is worst than the best model. We analyse in detail each model performances. 3.2
In order to analyse each model results, we depict in figure 3 the histograms of the outputs of each model M1, ..., M4 on the test set. The shape of each histogram largely differs from one model to another. We then investigate a simple statistics that may indicates from this shape the quality of the model.
A detailed analyse of central and extreme values of these histograms reveal that for the best model, the center (bins 5 and 6) is bigger than the extremities (bins 1 and 10). We then compute a simple ratio:
Q = h(center)/h(extremities), where h is the histogram here of 10 bins, so h(center) = h(5) + h(6) and h(extremities) = h(1) + h10).
Q is high if the border estimates of a model are rare, that is if the model if ’careful’ (most of the decision are close to the decision boundary). Thus we call this index the ’Carefulness Index’.
In figure 4 we give the log(Q) values and the AUC results for each of the four models. We see that when Q decreases, AUC is also decreasing, moreover the ranks given by Q are similar to the AUC ranks. 4
Depicting the results of very different models we enlighted a simple statistics on the raw model outputs that seems to be tied to its performances. The very different models we tested have different carefulness index. The experiments show that more careful is a model, more it AUC increases. This result shall be confirmed on other raw distributions of other model outputs. This kind of global shape statistics are interesting for scaled systems, where fast and unsupervised estimates of visual detector quality shall be possible. Further work will be conducted in this field in the AVEIR group.
This work was supported by French National Agency of Research (ANR-06-MDCA-002). [1] Joachims, T.: A Support Vector Method for Multivariate Performance Measures, In Proceedings of the International Conference on Machine Learning (ICML) (2005) [2] Fakeri-Tabrizi, A., Tollari, S., Denoyer, L., Gallinari, P.: UPMC/LIP6 at ImageCLEFannotation 2009, Large Scale Visual Concept Detection and Annotation, In CLEF working notes 2009 (2009). [3] VandeSande, Gevers T., and Snoek C.: Evaluation of Color Descriptors for Object and Scene
Recognition, In Proceedings of CVPR. Anchorage, Alaska, USA (2008) [4] Mulhem et al.: MRIM-LIG at ImageCLEF2009 photo annotation, In CLEF working notes 2009 (2009) [5] Zhao, Q., Glotin, H. and Dumont, E.: LSIS Scaled Photo Annotations - Discriminant Features
SVM vs Visual Dictionary based on Image Frequency, In CLEF working notes 2009 (2009) [6] Ferecatu, M. and Sahbi, H.: TELECOM ParisTech at ImageClef 2009: Large Scale Visual
Concept Detection and Annotation Task, In CLEF working notes 2009 (2009) 0 . 3 −0.5 −1 1 2 3
4 Models Figure 4: The relation between Q index and the AUC for the four models. Log(Q) are the positive (blue) values, while the negative (red) are the log(AUC). We see that when the carefulness index Q decreases, AUC is also decreasing, moreover the ranks given by Q are similar to the AUC ranks.