This paper presents the participation of our group in the ImageCLEF 2012 Photo Annotation Task. Our approach is based on visual and textual features as we experiment with different strategies in order to extract the semantics inside an image. First, we construct a textual dictionary of tags using the most frequent words present in the user tag annotated images from the training data sets. A linear kernel is then developed based on this dictionary. To gather more information from the images we further extract local and global visual features using TopSurf and Profile Entropy Features as well as Color Moments technique. We then aggregate these features with Support Vector Machines classification algorithm and train separate SVM models for each concept. In the end, to improve our system's performance, we add a postprocessing step that verifies the consistency of the predicted concepts and also applies a face detection algorithm in order to increase the recognition accuracy of the person related concepts. Our submission consists of one visual-only and four multi-modal runs. We further give a more detailed perspective of our system and discuss our results and conclusions.
ImageCLEF 20121 Photo Annotation Task2 represents a competition that aims to
improve the state of the art of the Computer Vision field by addressing the problem of
automated image annotation [
In 2012, the organizers offered a database consisting of 15,000 training images annotated with the corresponding 94 binary labels and a set of 10,000 test images which were to be automatically annotated (see Figure 1). The images were extracted from Flickr3 online photo sharing application and so each image had the associated EXIF data and Flickr user tags. Among the reasons that make this task of image annotation a difficult one are the diversity of the concepts simultaneously present in an image, the subjectivity of the existing annotations in the training set (especially regarding the feelings related concepts) and the fact that the training samples are unbalanced and so there may be more examples for a concept then for another.
The system we propose combines different state of the art image processing
techniques (TopSurf, PEF, Color Moments) with Support Vector Machines and
Kernel functions we defined in an attempt to obtain good overall performances. This
was our second participation in Photo Annotation task, after our contribution from
2009 [
The rest of the article is organized as follows: Section 2 presents the visual and textual features we extracted to describe the images, Section 3 covers the classification and post processing modules of our system, Section 4 details our submitted runs and Section 5 outlines our conclusions.
TopSurf4 [
The TopSurf library assigns Tf-Idf scores [
1 ∗ 2 | 1| | 2| = ∑ ∑ 1 1
2 ∑ 2
The similarity score will be between 0 and 1 (because the angle of the vectors d1 and d2 is smaller than 90 degrees), with 1 for identical descriptors and 0 for absolutely different ones. The time needed to compare these descriptors is, on average, 0.2 ms (with a database of 100,000 images).
Profile Entropy Features (PEF) [
The PEF features are computed on an image I by using the normalized RGB
channels: = , = , ! = 1 − − , where # = $&$%. The profiles of the
orthogonal projections of the pixels to the horizontal X axis is noted ' ( and to the
)
vertical Y axis (' (), where op is the projection operator (arithmetic or harmonic
*
mean). The length of a profile is + = , or + = - , (where , denotes I’s
columns and - , denotes I’s rows) and we estimate its probability distribution
function (( . ) on / = 01 √+ bins [
The algorithm repeats for each of the 3 equal horizontal sub-images (see Figure 2) and on the whole image. The PEF descriptor is denoted by the concatenation of 4567, 456?, 456% the mean and variance of the 3 channels, thus we have 4 regions × 5 features × 3 channels = 60 dimensions that describe the image I.
Color moments represent a method that can be used to differentiate images based on
their features of color. The main idea behind color moments is the assumption that the
distribution of color can be interpreted as a probability distribution, which can be
characterized by a number of moments (mean, variance, etc.). Stricker and Orengo
[
A color can be abstractly represented by using color models like RGB (Red, Green, and Blue) or HSV (Hue, Saturation and Value). Thus, each of the three dimensions of the chosen color model is characterized by three moments of a color distribution, resulting in a nine dimension vector which will describe the color distribution in a given image.
5 = ∑B ( B, 5 is the mean or the average color value in the image, ( B is the value of the jth pixel in the ith dimension of the color model and N is the number of pixels in the image.
C = D ∑B C is the standard deviation (the square root of the variance) of the distribution.
E = D ∑B
( B − 5 &, is the skewness of the distribution which is a measure of the degree of its asymmetry.
The similarity function F9F can be used to adjust the weights (G ) of each channel, because it makes sense that, for example, the hue of a color is more important than its intensity. The function is defined as the sum of the weighted differences between the moments of the two distributions:
F9F , , , ∑&
G |5 " 5 | H G |C " C | H G | " |
In some situations, the visual information is not enough to give a semantic interpretation of an image and this is why we exploit user defined tags to improve the judgment of the whole system. The problems that arise with these approaches are the fact the number of user defined tags is relatively small (or 0), the tag can be in any language, some of them are irrelevant or they are a concatenation of words (see Figure 3). These problems make the traditional methods used in the field of natural language processing inapplicable in this situation.
The authors in [
I JK , KBL = K MKB
K MKB is the dot product between the transposed binary vector K and the KB vector. The KG kernel counts the number of shared user tags between two associated images.
Support vector machines [
We propose a combined SVM kernel that makes use of all the features described above:
IN9FO PQ@ R, S = TUVIUV R, S + T:QAI:QA R, S + TWUIWU R, S + TNFINF R, S
Where TUV, T:QA, TWU, TNF ∈ [
IUV R, S = UV R , UV S is the cosine similarity defined in section 2.1 for the TopSurf library; I:QA R, S = exp −_||R − S|| ) is the RBF kernel and it is used with PEF descriptors (section 2.2); IWU(R, S) = U(`)aU(b) is the linear kernel defined in section 2.4 normalized by
P the number of tags in the dictionary; INF(R, S) = exp(−_ F9F(R, S)) is the kernel that uses F9F function for color moments (section 2.3) and _ is the regularization parameter.
These functions and Kd=efghij kernel satisfy Mercer’s theorem [
In the post processing module of our system we ensure that the classifications made by SVMs models are correct. For example, if an image is classified with quality_noblur and quality_partialblur at the same time, we adjust the concept’s probabilities so they sum up to 1. We learn about mutual exclusive concepts ( 5R ) from the training set. Let +k be the set of predictions made by SVMs, with T , T ∈ 5R and T , T ∈ +k: +k = +k \ mT ∶ (T , T ) ∈ 5R, ( (T ) o ( (T )p
We also compute the Voila – Jones face detection algorithm [17], in order to count the number of persons in a given image (the concepts regarding the number of persons in this year’s competition are: quantity_none, quantity_one, quantity_two, quantity_three, quantity_smallgroup, quantity_largegroup) and to determine if view_portrait concept is present.
Our system (Figure 4) has a modular and flexible structure and can easily be extended with some other feature extractors’ algorithms: We participated at this year ImageCLEF 2012, Photo Annotation Task by submitting 5 runs with different configurations: • Submission1: Visual only configuration with the following parameters: TUV = 0.6, T:QA = 0.2, TWU = 0.0, TNF = 0.2 and SVM’s regularization parameter: C = 20 with post processing step; • Submission2: Multimodal run with the parameters: TUV = 0.45, T:QA = 0.1, TWU = 0.35, TNF = 0.1 and SVM’s regularization parameter C was chosen separately for each of the 94 classifiers, with sampling for some of the concepts; • Submission3: The same configuration as for Submission2, with the sampling strategy applied for each of the 94 classifiers; • Submission4: Multimodal run with the parameters: TUV = 0.35, T:QA = 0.25, TWU = 0.25, TNF = 0.15 and SVM’s regularization parameter C was chosen separately for each of the 94 classifiers, with the sampling strategy applied for each of the 94 classifiers; • Submission5: Multimodal run with the parameters: TUV = 0.45, T:QA = 0.1, TWU = 0.35, TNF = 0.1 with SVM’s regularization parameter C = 20 and without the post processing step.
Our best run was the one with Submission1 configuration and it was ranked 11th of
a total of 18 group participants [
All participants at ImageCLEF 2012 in Photo Annotation task have submitted several runs using not only visual strategies based on features extracted from the images but as well textual ones based on user defined tags that were given alongside the images. The best results however, as it can also be observed from the table above, were achieved by the systems that managed to combine both the visual and textual features together. What our system lacked was the fact that we did not find the best balance between feature extraction algorithms (with their contribution in the learning step) and also the fact that some of them should weight more or less depending on the concept that is being learned.
In this paper we combined several different state of the art algorithms for image processing together with Support Vector Machines and kernel functions in order to approach the task of automated image annotation. As images can be annotated with more than one concept we tried to increase our system’s performance by using not only local image feature descriptors (TopSurf), that for example, proved to be unpractical at detecting feelings in an image, but also try analyzing the colors (Color Moments) and the textures (Profile Entropy Features) in the image and even make use of the user defined tag semantics and face detection algorithms.
All experiments were made using the approach we presented in this paper and careful attention was given to the selection of the threshold parameters of the SVM kernel function that we used, IN9FO PQ@, TUV, T:QA, TWU and TNF.
As future work, we will try and set different values for these parameters taking into consideration the concept that the classifier is training for. For example, for concepts that express feelings, Color Moments technique should have the deciding weight, whereas for panoramic images a greater weight should be given to the texture descriptor (PEF).
Acknowledgement. The research presented in this paper was funded by the Sector Operational Program for Human Resources Development through the project “Development of the innovation capacity and increasing of the research impact through post-doctoral programs” POSDRU/89/1.5/S/49944. 15. Gidudu, A., Hulley, G., Marwala, T.: Image Classification Using SVMs: One-against-One Vs. One-against-All. In Proceeding of the 28th Asian Conference on Remote Sensing, Malaysia, CD-Rom. (2007) 16. Witten, I., Frank, E., Hall, M.: Data Mining – Practical Machine Learning Tools and
Techniques, Third Edition. Morgan Kaufmann, 629 pages. (2011) 17. Viola, P., Jones, M.: Rapid object detection using boosted cascade of simple features.
Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518, Kauai, Hawaii, USA. (2001)