This report describes our experiments in the Content Selection subtask of the Image Annotation task of ImageClef 2016[7, 13]. Our approach is based on the fact that the human visual system concentrates mostly on local features [12]. In this respect, we trained an SVM classier with descriptors that are based on the local features of the image, such as edges and corners. For the experimentation process we used the set of 500 images provided for the task, divided into training and test set. This set was particularly created for this year's new subtask, Content Selection, although the concepts are the same as last year. Through experimentation we determine which descriptors give the best results for the given task. To conduct the main experiment the SVM classi er is trained with the aforementioned set of 500 images using a subset of the top-performing features. Consecutively, the SVM processes the new set of 450 images and selects the boxes that best describe them conceptually.
Content Selection is the intermediate step between identifying objects of an image and generating a natural language caption. The objective of this task is to identify which bounded objects of the image are important so as to be included in the annotator's caption. In addition, since each object is labeled, the result would be a conceptually accurate description of the image. As requested by the ImageCLEF competition we developed a system that receives as input labeled objects of an image and identi es the objects that are referred to in the corresponding image caption. Since participants of the content selection subtask have concentrated so far on using only text features or bounding box information, this paper may provide a novel contribution in exploring features based on visual keypoints only. We are not aware of another work with keypoints being used in our ways (e.g. ratio of keypoints in bounding boxes to keypoints in image).
Our approach relies on nding suitable descriptors, from the given data, in order to train an SVM classi er. In this work we followed an image-only approach without processing the textual information provided as groundtruth. Inspired by the fact that the human visual system concentrates mostly on local features, we incorporated several such descriptors using well known algorithms of image processing. Thereby we created a set of 17 descriptors and grouped them into several subsets referring to similar features. Then, after experimenting with descriptors standalone and as groups, we arrived to a set of 9 most-useful descriptors.
Complementary to testing for di erent feature subsets, experiments were also conducted using various SVM kernel functions. Linear, polynomial and Gaussian radial basis function kernels were tested. Polynomial and Gaussian kernels performed similarly, but the latter one was chosen as it produced slightly better results.
The rest of this report is organized as follows. In the next section we describe the dataset provided as well as the methodology we followed to tackle the problem. Speci cally, we analyze the descriptors that were used during experimentation and describe the SVM classi er. Section 3 describes the evaluation methodology along with our the experimental results. Conclusion and directions for further research are summarized in Section 4. 2
2.1
Dataset
The dataset provided by the ImageCLEF competition consists of 500 images
from various concepts, accompanied with a set of bounding boxes for each image
and a label for each box. Furthermore, a set of textual descriptions is given for
each image as ground-truth with a minimum of 5 and a mean of 9.5 sentences
per image [
Feature Extraction Initially, 17 descriptors were created. After experimentation we concluded to a subset of 9 that was found to have the maximum contribution. Correlation information between the descriptors and the purpose of each one was used to cluster them into categories. In the rest of this section, we will elaborate on each descriptor category.
Position Desctiptors For each bounding box, two points are given that de ne its position in the image. The coordinates of these points are used individually as four values xmin; xmax; ymin; ymax, divided by the corresponding dimension of the image, as follows: where w; h denote the width and the height of the image, respectively. Additionally, two more features are formed that correspond to the relative position of the center of the bounding box, abscissa and ordinate respectively: d1 = d2 = d3 = d4 =
The purpose of those descriptors is to investigate the correlation between the
position of the bounding box and the importance of it [
Size Descriptors The descriptors of this group aim to calculate the portion of
the image that is occupied by the bounding box, separately for the two
dimensions and combined as well.
where wb; hb denote the width and height of the bounding box, respectively.
Descriptors based on Local Features For the calculation of the descriptors
of this group we used several well-established algorithms for local feature
detection (i.e. the Canny edge detector [
Entropy Descriptors Image entropy is a quantity that is used to describe
the amount of information that is contained in an image. In this regard, the
motivation behind this feature group is to quantify the amount of information
held by the content of a bounding box, proportionally to that of the image.
As a rst step we produced three di erent versions of the image. The rst two
correspond to the edge maps generated by the Canny edge detector [
d15 = d16 = d17 =
E(x) =
N X h(i) log2 h(i) i=1 where E(x) denotes the entropy of an image x, de ned as where h(i) is the count of pixels assigned to the ith bin of the image histogram and N the total number of bins. 2.3
Support Vector Machine (SVM) Classi cation
For the purpose of this task, we trained a binary SVM in order to classify each
bounding box as `important' or `not important'. SVM tackles the problem of
nonliner classi cation by determining a hyperplane that separates two classes in a
space of higher dimensionality than the feature space using a kernel function [
During the experiments, the training dataset was divided into two subsets, the training and the validation subset, respectively. We experimented with training subsets of 100 and 250 images randomly selected. We investigated the performance of di erent sets of descriptors, concluding to a set of 9. Table 1 presents the results of the experiments with di erent con gurations.
In the cases where the SVM classi ed a small number of the boxes as important, we used the SURF descriptor as criterion for the selection. Speci cally, if the result contained less than two boxes, then the n2 + 3 boxes with the biggest d13 value were added to result set, where n is the number of boxes in the image. In case the image contained less than n2 + 3 boxes, then all of them were selected.
The number n2 + 3 was selected after experimentation. We initially started with number n2 and then concluded to n2 + 3. 3 3.1
Evaluation Measures According to the instructions, Subtask 3 is evaluated using the content selection metric, which is the F1 score averaged across all test images. Each F1 score is computed from the precision and recall metrics averaged over all gold standard descriptions for the image.
The precision P Ii for test image Ii is computed as: The recall R(Ii) for test image Ii is computed as:
P Ii = RIi = I = fI1; I2; : : : ; IN g the set of test images GIi = fGI1i ; GI2i ; : : : ; GIMi g the set of gold standard descriptions SIi the resulting set of unique bounding box instances M the number of gold standard descriptions for image Ii.
The content selection score F Ii for image Ii, is computed as:
F Ii = 2
P Ii RIi
P Ii + RIi The nal P , R and F scores are computed as the mean P , R and F scores across all test images. (1) (2) (3) 3.2
Experimental Results For the experiments we used the 500 images from imageCLEF dataset. Firstly, the images are split randomly in two sets, the training set and the test set. As training set, 100 or 250 images are used, 20% of 50% of the set respectively. As expected, the bigger training dataset gave better results. for this reason, it was decided to use all the 500 provided images dataset as training set at the submission run. Experiments took place with various combination of descriptors. As shown in Table 2, descriptors are managed as groups. For decision criterion in the cases that the SVM classi ed a small number of boxes as important, we experimented with d9 and d13 descriptors. The SURF descriptor (d13) produced better results, as it is a well-established and robust algorithm, in contrary to descriptor d9 which is more abstract. Table 2 shows the setup of each experiment. For example, for experiment 1 descriptors d1:::4, d11, d13 and d17 were used. SVMs kernel is radial basis function, 100 images of 500 were used as training set and as criterion d9 were used. Before the experiments each group of descriptors were tested separately, so the behavior of each was known. That means that there was not need to include or exclumide all descriptors of the group in each experiment but the best representatives of the group. Furthermore, the correlation matrix was taken into account, that is why for example d9 is not included in our experiments, as shown in Table 2. Finally, as seen from Table 1, the Fmeasure was not the only one taken under consideration for our nal choice but also the balance of precision and recall.
Our approach using Gaussian kernel SVM classi er and 9 descriptors as experiment 14 achieves an overall F-measure of 54:59% 15:33, the best result of the two participating groups to the subtask. This report describes the methodology, the experimentation, and the results acquired concerning DUTh's participation to the Subtask 3 of the Image Annotation task of ImageCLEF 2016. Our novelty is that we have tackled the task using only visual keypoints/features, in contrast to other participants so far using only text features or bounding box information. We investigated the performance of seventeen image descriptors combined with three SVM con gurations corresponding to di erent SVM kernels. Experimental evaluation highlighted the signi cance of a feature subset in determining the conceptually important boxes. These features mostly relate to the edges and corners detected by wellestablished algorithms. Our approach using Gaussian kernel SVM classi er and nine descriptors achieves an overall F-measure of 54:59% 15:33, the best result of the two participating groups to the subtask.
Taking a step further, we believe that the proposed methodology can be
improved towards two directions. The rst one concerns the improvement of
the feature extraction methods. Motivated by the high performance of local
feature detection demonstrated by this project, we would like to additionally
incorporate the local feature descriptors that correspond to them. The statistical
analysis and the comparison of these descriptors may provide useful information
concerning the importance of each key-point. Room for improvement also exists
in the exploitation of textual analysis of the proposed annotation terms. Textual
features that are based on term and document frequencies can provide useful
insight in order to determine the importance of every box label. Furthermore,
textual features concerning term signi cance may be extracted exploiting word
ontologies or semantic networks, such as WordNet [