This working note describes the participation of TIA-INAOE in the Photographic Retrieval and the Large Scale Image Annotation tracks at ImageCLEF2009. We developed speci¯c methods for each track with the goal of exploiting the information available while maximizing annotation and retrieval performance. On the one hand, for the retrieval track, we proposed a post processing technique for re-ranking documents according to di®erent diversity categories. With this formulation we considered both visual and textual features and we incorporated information of the di®erent categories by which topics are clustered. Results obtained with this technique suggest it is a promising method for result diversi¯cation. However, we still need to deal with issues that a®ect the retrieval performance. On the other hand, for the annotation task, we adopted a simple annotation technique based on KNN classi¯cation. Only global features were considered under this formulation. The output of the KNN method was then re¯ned by means of an energy-based model that attempts to maximize the semantic cohesion among labels assigned to each image. We considered information of the annotation hierarchy to constraint certain labeling con¯gurations. Results obtained with this technique give evidence that the KNN approach is an e®ective annotation technique despite being rather simple. The re¯nement strategy resulted useful for improving the labeling performance of an ine®ective annotation method; although we could not improve the labeling performance of a strong baseline. Summarizing, the results obtained at ImageCLEF2009 are encouraging and motivate further research in several directions that we are currently exploring.
This working note describes the participation of TIA-INAOE in the Photographic Retrieval and the Large Scale Image Annotation tracks at ImageCLEF2009. A total of 10 runs were submitted that comprise di®erent settings of our proposed methods for image annotation and retrieval. The proposed methods aim exploiting the information available while maximizing annotation and retrieval performance.
On the one hand, for the retrieval task, we adopted a two stages retrieval process. In the ¯rst stage, an initial image search is performed by using only textual information to obtain potentially relevant documents. In a second stage the candidate documents are re-ranked by considering the di®erent diversity categories (clusters) as provided by the organizers. For re-ranking we considered both visual and textual information. Results with this approach are mixed: whereas the reranking technique resulted helpful for diversifying retrieval results, the best retrieval performance (MAP) was obtained with the baseline retrieval method. This result suggests that textual retrieval methods are better suited for this collection.
On the other hand, for the annotation task, we adopted a three stages methodology. In a
¯rst stage, candidate labels for test images are selected with a KNN approach, which consists of
obtaining the labels assigned to the K-nearest neighbors (in the training set) of the test image. In
the second stage, an energy-based model is used to select, among the candidate labels, the disjoint
labels for the image, see [
The rest of this document is organized as follows. In the next section we describe the approach we adopted for the photographic retrieval task and the results obtained with this technique. Next, in Section 3, we present the annotation method we proposed as well as o±cial results of this method. Finally, in Section 4, we describe the conclusions derived from this work and we outline future work directions. 2
We proposed a two stage approach for the photographic retrieval task at ImageCLEF2009; our
methodology is depicted in Figure 1. In the ¯rst stage a set of m potentially relevant documents
is retrieved by using a text-based image retrieval technique. In the second stage the m documents
are re-ranked by taking into account the initial score assigned to documents and the similarity
of candidate documents to the diversity clusters provided by the organizers. We proposed this
formulation for two main reasons: 1) we wanted to take advantage of the topic clusters for
diversifying retrieval results and 2) we wanted to make more e±cient the search process. The latter was
accomplished because the initial search can be performed e±ciently (over the 500; 000 documents)
and once the set of potentially relevant documents is reduced to a subset of documents we can
compare images in acceptable time and we can apply more complex strategies over this reduced
set of documents. The rest of this section describes in detail our method. Further details on the
task and on the collection are described by Lestari et al. [
The proposed method considers both textual and visual information for representing documents. As textual features we consider a tf-idf weighting scheme over words for representing documents, see the next section. As visual features we use color histograms on both RGB (256 bins) and HSI (128 bins per channel), for a total of 640 visual attributes. We also performed preliminary experiments with other visual features, including: edge histograms, color histograms in CIE-Lab, texture features and local descriptors; however, the RGB and HSI color histograms resulted more e®ective (according to an empirical evaluation we conducted) for retrieving images under the Euclidean distance. 2.2
For the initial retrieval we considered the vector space model (VSM) under tf-idf weighting for
representing documents [
We considered textual information only, because computing the Euclidean distance, between query images and the 500; 000 images that compose the collection, would be very computationally expensive. Thus, the re-ranking approach was also used for e±ciency reasons (query images and a reduced set of m¡images can be compared in acceptable time).
For querying we used all of the textual information available in topics (i.e. title, cluster titles and cluster descriptions); this is motivated because at this stage we wanted to retrieve documents that were related to the query as a whole, so that in the next stage the search can be re¯ned. We ranked the documents by their similarity to the query and we keep the m = 1; 000 top ranked documents for the second stage; our baseline run consists of returning these 1; 000 documents. 2.3
In the second stage we re-ranked the m¡documents obtained by the initial retrieval method. For each topic category (cluster), j, we assigned a score to each of the m¡documents, di2f1;:::;mg, as follows: scfXinal(diX ; qjX ) = ¸ £ scinitial(di) + £SX (diX ; qjX ) (1) where scinitial is the similarity score obtained from the initial retrieval stage and SX (diX ; qX ) is an estimate of the similarity between the ith¡document and the jth¡sub-query under modality X. ¸ is a scalar weighting the contribution of the ¯rst term. A sub-query qjX is the part of the topic corresponding to the jth diversity cluster, where j 2 f1; : : : ; Cg and C the number of categories associated with the topic. The superscript X indicates which information modality is used: X = T means that textual information was considered (e.g. cluster title), X = V indicates that visual information was used (i.e. cluster image) and X = M means that both textual and visual information wee considered. When X = T , we used the cosine similarity as ST ; when X = V we used the (normalized) inverse of the Euclidean distance as SV ; when X = M we used SM = wm1 £ ST + wm2 £ SV , where the scalars wm1 and wm2 weight the contribution of each modality.
For each category j, the score assigned to the m documents was sorted in descending order; thus for each category we had a di®erent ranking for the m¡documents. The C¡rankings were combined (by means of round robin) to generate a ¯nal ranking for the m¡documents. The top 1; 000 documents according to the ¯nal ranking were submitted for evaluation. For the topics that do not have textual information (i.e. topics 25 to 50) we used the provided query images. 2.4
Textual retrieval method, see Section 2.2.
The m¡documents obtained by the initial retrieval method are reranked according to the score in Equation (1); although the topic was not separated into categories under this formulation.
Re-ranking technique with X = V , see Section 2.3.
Re-ranking technique with X = T , see Section 2.3.
Re-ranking technique with X = M , see Section 2.3.
The obtained results are mixed. The best retrieval performance, in terms of MAP and RP, was obtained with the baseline (i.e. a textual retrieval technique); P20 was higher with the R-5 con¯guration. Nevertheless, the di®erence in retrieval performance between the baseline and the other runs was of less than 0.03.
In terms of results diversi¯cation (i.e. C20), it is observed an improvement over the baseline for all of the runs (rows 3-6 in Table 2). The largest improvement in C20 was obtained with the methods that considered visual information (i.e. R-3 and R-5). The run R-3, which used only visual information for re-ranking documents, resulted particularly helpful for diversifying retrieval results (this run was ranked 67 out of 84). These results suggest that the re-ranking technique can be helpful for diversifying results and that using di®erent modalities for the initial search and the re-ranking technique results in better performance. Note that the performance of the reranking technique depends on the initial retrieval, thus we expect better diversi¯cation of results when better retrieval methods are considered for the initial search. We are studying this research direction. Retrieval performance is slightly a®ected by applying the re-ranking method. However, the performance of the initial search method was rather limited: this method was ranked 47 out of the 85 submitted runs. 2.5
The results obtained by the TIA-INAOE team in the photographic retrieval task at ImageCLEF2009 may seem discouraging in a ¯rst instance. However, interesting ¯ndings can be drawn from our participation: the proposed re-ranking technique resulted helpful for result diversi¯cation, although it slightly a®ects the retrieval performance; better diversi¯cation performance was obtained when the re-ranking was based on visual information only; better diversi¯cation performance is expected if a better search engine is used for the initial retrieval; as whole, the proposed formulation can be helpful for e±cient multimedia retrieval in large scale image collections. 3
We proposed a three steps methodology for facing the annotation task at ImageCLEF2009. For each test image, we identi¯ed a subset of candidate labels by comparing the test image to the training ones. Then, we selected the disjoint labels for the image by means of an energy-based model. Next, optional labels were assigned by taking into account co-occurrence statistics.
The method described in this section is based in the assumption that similar images have
associated similar labels. Thus, for each test image we considered the labels assigned to the K¡most
similar training images; then, we applied di®erent strategies for selecting disjoint and optional
labels for a test image. The bene¯ts of adopting this methodology are annotation e±ciency,
implementation simplicity and the competitive performance that can be obtained with the proposed
formulation. The rest of this section describes our methodology and the obtained results. Further
details on the task and on the collection are described by Nowak et al. [
We used global features to represent images. In particular, we considered an RGB color histogram (256 bins), texture features extracted from the co-occurrence matrix (88 values), an edge histogram (360 bins) and an HSI color histogram with 128 bins per channel for a total of 1; 088 attributes. Each image was represented by its vector of features; thus, hereafter, we will refer as images to both the images themselves and the vectors of features representing the images. For comparing images we used a weighted Euclidean distance, where a di®erent weight is used for each subset of features (i.e. RGB, texture, edge, HSI). The weights were set empirically by trial and error. 3.2
The ¯rst step in our methodology (depicted in Figure 2) is to obtain the k¡most similar trainingimages to each test image IT , we denote this set of images by INTN . We are interested in the labels associated with the images in INTN , we denote the corresponding set of labels by LTNN and we call it the set of candidate labels for IT . We call the positions of label liT to the set of positions, in the sorted set INTN , occupied by images that have liT as annotation. Then, we assigned a score to each label liT 2 LTNN as follows: ª(liT ) = ®r £ wr(liT ; LTNN ) + ®a £ wa(liT ; LTNN ) + ®r £ ws(liT ; LTNN ) (2) where wr(li; LTNN ) is the normalized frequency of occurrence of label liT in LTNN ; wa(li; LTNN ) is the average of positions of label liT ; ws(li; LTNN ) is the standard deviation of the positions of label liT ; ®r, ®a and ®s are scalars that weight the contribution of each term into the ¯nal score. 1.4 1.2 1
The scores assigned to the candidate labels were used by the model to be described below.
Alternatively, a rather simple labeling approach consists in sorting the candidate labels, in
descending order of ª(li) (Figure 2, right), and using the top t¡labels for annotating the test image
IT . We call this setting our baseline1 run. This annotation approach is based in the work of
Makadia et al., where the labeling problem is faced as one of retrieval [
We also considered an alternative re-ranking approach that aims re¯ning the ranking of the
candidate images (as obtained with global attributes) by considering local features. Under this
technique, we obtained the k0¡most similar images (k0 > k) to each test image, using the features
described above and the Euclidean distance. Next, we re-ranked this k0¡images by using a naijve
Bayesian classi¯er (NBC). The NBC evaluates the pixel similitude between a patch of the image
in the test set, with several patches of the k0 images, using the Euclidean distance as well. Patches
were obtained from little regions in the image that were passed through Gabor and max ¯lters.
This is according to a simpli¯ed Bayesian approximation of a bio inspired model of the visual
cortex [
Once we identi¯ed a set of candidate labels for a test image, as described in Section 3.2, we applied a labeling re¯nement method for selecting the disjoint labels for the image. Intuitively, we wanted to select, from the set of candidate labels, the best combination of disjoint labels, using co-occurrence statistics calculated from the training set.
1Note that we have applied the same postprocessing described below for selecting labels under this formulation.
We de¯ned an energy-based model for the disjoint categories (i.e. Seasons, Place, TimeOfDay, Illumination, Blurring and Persons) using a variable aj per category, with j 2 f1; : : : ; 6g. Each random variable can take values from its corresponding set of possible labels (e.g. the variable corresponding to the category TimeOfDay, can take the values: Day, Night or No-Visual-Time); we denote the assignment of label lx to variable aj with ajx, thus ajx can be considered a label itself. Additionally, we restricted the values that each variable can take, by considering as possible values only to those labels that appear in the set of candidate labels (i.e. LTNN ). Figure 3 depicts the modeling process for a particular test image.
The goal of the model is to select the con¯guration of labels A (i.e. a label assignment per category) that maximizes the cohesion among the labels assigned to the image. Accordingly, we assigned an energy value to each con¯guration of labels as follows:
E(A) = ¡¡ X ± £ ª(ajx) + £ X
X
½(ajx; ayh)¢ aj2A aj2A ah2´aj
Where ª(ajx) as in Equation (2) and ½(ajx; ayh) is a factor that weights the association between labels lx and ly, assigned to categories aj and ah, respectively; ´aj is the set of neighbors2, under the model, of category aj , as we are using a fully complete graph the set of neighbors for aj is ´aj = ap 2 A : p 6= j. ± weights the contribution of the initial ranking to the energy of the con¯guration. We used co-occurrence statistics to estimate the association between labels. Speci¯cally, we estimate ½(ajx; ayh) as follows: ½(ajx; ayh) =
#(lx; ly)
#(lx)#(ly)
where #(lx) is the number of images in the training set in which label lx occurs and #(lx; ly) is
the number of images in which both lx and ly co-occur. For speci¯c labels lx and ly, the higher
½(ajx; ay ) the more both labels are associated. Note that ½(ajx; ay ) can be calculated for any pair
h h
of labels (disjoint and optional); thus we use this association information in the next section for
2Note that these neighbors are di®erent from the neighbors considered in Section 3.2.
(3)
(4)
selecting optional labels as well. Equation (3) assigns low energy values to correct con¯gurations
and large values to incorrect ones. Therefore, the problem of selecting disjoint labels for a test
image is reduced to that of ¯nding the con¯guration of labels that minimizes Equation (3); for
this work we used iterated conditioned modes (ICM) for this task [
The energy-based model returns labels for each disjoint category. Intuitively, the model selects
the combination of labels that maximizes their semantic cohesion. This method is based on the
method proposed in [
Once we selected the disjoint labels we assigned optional labels to each test image as follows. We assigned a score to each candidate label liT (that does not belong to any disjoint category), identi¯ed in Section 3.2, as follows: ³(liT ) = ª(ljT ) £ 6 Y ½(liT ; ldj ) j=1 (5) where ª(ljT ) and ½(liT ; ldj ) are de¯ned as above and ldj is the label assigned to the jth disjoint category. We ranked labels according to ³(liT ) and used the top¡n labels for labeling the test image. 3.5
For generating the ¯nal annotation for a test image we applied the following postprocessing. First, regarding the number of labels, we assigned the top 4 optional labels to each test image, as 4 is the average number of optional labels that were used for annotating the training images. Second, when a leaf-label was chosen as optional label for an image, we also included its parent label, as appear in the annotation hierarchy de¯ned by the organizers. Thus, for example, if the label Lake was considered as optional label for the image, we also included the label Water. Of course, this is only applicable to optional labels that appear in the hierarchy as leafs. 3.6
Table 3 summarizes the ¯ve runs we submitted for the large scale annotation task; whereas Table 4
shows the results obtained by such runs. We show the following performance measures: the
hierarchical measure described in [
From Table 4 we can see that the baseline method is a rather strong baseline, con¯rming the
results reported by Makadia et al. [
The worst performance, in terms of H-A and H-WA, was obtained when local information was used for ranking labels according to the KNN approach. Hence these local features were not helpful for re-ranking. An interesting result, however, is that when the energy-based model was applied with the labels ranked according to local features, the energy-based model was able to improve the performance of the former signi¯cantly (compare the performances of runs A-2 and A-4). This result suggest the energy-based model can be helpful when the initial labeling is not good.
ID A-1 A-2 A-3 A-4 A-5
KNN; the score in Equation (2) is used for assigning labels.
KNN-RR; we use the alternative re-ranking with local features (see Section 3.2.1) and the score in Equation (2) for assigning labels.
KNN + EBM; candidate labels selected as in A-1, the energy-based model is used to select the disjoint labels and optional labels are selected as described in Sections 3.3, 3.4 and 3.5 KNN-RR + EBM; candidate labels selected as in A-2, the energy-based model is used to select the disjoint labels and optional labels are selected as described in Sections 3.3, 3.4 and 3.5 KNN-RRW + EBM; same as A-4, but we use a larger ± value.
It is interesting to note that whereas the performance of our runs in H-A and H-WA was, to some extent, satisfactory, the performance in terms of EER and AUC was rather limited. Our best run (A-1) was ranked 26 out of 74 submitted runs in terms of H-A and H-WA. However, the same run was ranked 59 and 66 out 74 in terms of EER and AUC, respectively. This result suggest that the method can label images as whole satisfactorily, although its per-label performance is limited; which is not surprising as we have not developed visual concept detectors per class. Note that the main goal of assigning labels to images is to support annotation based image retrieval methods, which use labels assigned to the images as a whole. Thus, it seems that our method could support e®ectively this form of image retrieval, we will study this aspect as future work, currently we are conducting an more in-deep analysis of the results.
Finally, the processing time3 of our methods is quite acceptable, this time could be further reduced if we use software that is less computationally expensive (we used MatlabR for all of our experiments) and if we optimize our code. 3.7
The results obtained by the TIA-INAOE team are encouraging. The KNN approach to image labeling proved to be a very e®ective method for image annotation, regardless of its simplicity and generality. Despite the proposed energy-based model did not improve the performance of the KNN method, it was able to improve signi¯cantly the performance of the KNN-RR method. Thus suggesting the energy-based model can be helpful when the initial method is not very e®ective; this is a desired behavior of the model. Note that the energy-based model is still under development and that we have ¯xed the number of labels that are assigned to an image, also the parameter selection process can be improved. In general terms, our annotation methodology o®ers a good tradeo® in terms of annotation performance (in H-A and H-WA) and processing time. We would like to emphasize that the proposed approach for labeling re¯nement is not restricted to our KNN annotation method. It can be applied as a postprocessing step with any annotation method, 3The reported time does not include feature extraction from images. provided that the labels can be ranked, hence, showing the generality of the method and the potential impact it can have. Summarizing, the energy-based model is intuitively sound and is a promising method in which we are still working on. 4
We have described the participation of TIA-INAOE at ImageCLEF2009. Our team submitted runs for the Photographic Retrieval and for the Large Scale Annotation tracks. We proposed speci¯c methods for each track. On the one hand, we described a re-ranking approach that aims at maximizing the diversity of retrieved documents at the ¯rst positions. Our results show that whereas the proposed technique can improve the diversity of results, the base retrieval system still needs to be improved. On the other hand, we adopted a simple method for image annotation, and introduced a labeling re¯nement technique with the goal of improving the annotations as obtained with the former method. Our results suggest the KNN approach is e®ective for annotation and very e±cient. However, there are still several issues with our re¯nement method that we are currently working on.
Acknowledgements. The authors thank the organizers of ImageCLEF2009 because of their support. This work was partially supported by CONACyT under project grant 61335 and scholarship 205834.