This work describes the approach used by the RECOD team in the MediaEval Placing Task of 2013, in which we were required to develop an automatic scheme to assign geographical locations to images. Our approach is multimodal, considering textual and visual descriptors, which are combined by a rank aggregation strategy. We estimate the location of test images based on the coordinates of top-ranked images in the list of combined results.
Geocoding multimedia material has gained great attention
in the latest years given the importance of providing richer
services for users, like placing information on maps. Image
geocoding is the objective of the Placing Task in 2013, i.e.,
it requires participants to assign geographical locations to
images. Details about the Placing task, its dataset, and the
evaluation protocol can be found in [
In this paper, we present our multimodal approach that
combines di erent textual and visual descriptors uniformly.
We combine them using a rank aggregation strategy,
previously introduced in [
We handled the task of automatically assigning a geographical location to images using nearest neighbor searches on aggregated ranked lists, which combine textual and visual features. The strengths of our approach are its simplicity and its power to combine multiple description modalities.
For evaluation purposes in the training phase, we have selected a validation set of 5,000 images from the development set of around 8.5 million images. First, each photo from the development set was assigned to a xed cell of 1-by-1 degree based on its ground truth latitude and longitude. Then, the resulting grid was summarized by the total of photos (density) in each cell regarding to the dataset size. Finally, the evaluation images (5,000 photos) were randomly picked up from each cell, by taking into account its density.
From textual metadata, we used only the photo tags to compute similarities between the images. The tags were stemmed and stopwords were removed. The text similarity functions used were BM25 and TF-IDF, as implemented by the Lucene API.
Given the large dataset, we had to select carefully the
descriptors to be used. Initially, we have evaluated some of
the descriptors provided with the dataset, like: color and
edge directivity descriptor (CEDD), scalable color (SCD),
gabor lter. Using the validation set, we have noticed that
the best results were achieved by CEDD. Although SCD has
shown the best results in [
Additionally to CEDD, we used BIC (border/interior
pixel classi cation). This descriptor was chosen due to its
good results in large scale experiments [
As last year, we used a rank aggregation strategy to
combine di erent descriptors [
We have used an aggregation function similar to sima
(numerator is m instead of 2) proposed in [
For geocoding the test images, we have used a nearest neighbor approach. We used the development set ( 8.5 million images) as geo-pro les and each test image was compared to the whole development set. For comparing the images, we have used each type of feature independently (textual or visual). For a given test image, the ranked list of each feature is produced. All the lists are then combined by our rank aggregation strategy and the nal ranked list is generated. The lat/long of the rst image (most similar) in this nal list is assigned to the test image. 3.
Our submissions for this year are: run1: combines 2 textual descriptors: BM25 + TF-IDF; run2: combines 2 visual descriptors: BIC + CEDD; run3: one visual descriptor: BIC; run4: combines 2 textual and 2 visual descriptors: BM25 + TF-IDF + BIC + CEDD; run5: combines 4 textual descriptors: BM25 + TF-IDF1.
Runs 1 and 5 used only textual features. Thus, for test images without tags, there was no way to apply our similarity ranked list approach. As post-processing, we randomly selected an item from the development set to transfer its latitude and longitude to the test image. 3.1
Besides the organizers' standard evaluation metric, we
also applied the WAS score we proposed in [
Let d(i) be the geographic distance between the predicted and the ground truth location of the image i. The proposed score for the result of a given test image i is de ned as: score(i) = 1 lolgo(g1(+1+Rdm(ia)x) ) , where Rmax is the maximum distance between any two points on the Earth's surface (half of Earth's circumference at the Equator is 20,027.5 km).
Let D be a test dataset with n images whose locations need to be predicted. The overall score for the predictions of a method m is de ned as: W AS(m) = Pin=1 sncore(i) . 1st Quartile
Median As we can observe in Table 2, the test runs based solely on textual information yielded the best results (runs 1, 4, and 5), while those based only on visual descriptors presented low accuracy. The possible reason is the semantic gap, as there might be many di erent places with similar visual appearance, specially in a large dataset like the one used for training. Another potential issue was the large number of ties in the rst positions of ranked lists of visual descriptors. Given our 1-nn geocoding approach, this probably degraded our results. However, we can see that by combining 1Non-English tags were translated to English using the Google Translate service and combined with the original tags.
BIC+CEDD (run 2) we improve the results of BIC alone (run 3). The combination of textual and visual descriptors (run 4) was slightly worse than the textual descriptors isolated. One possible reason is the large di erence between textual and visual results.
Observe that for the test set (Table 2), our results were quite di erent for our validation set (Table 1), mainly for the visual features. While in the test3 set, BIC achieved less than 1% in the 1km radius, in the validation set, it presented 15.32%. Because of this, in the validation set, the fusion (run 4) results improved over run 1. The huge di erence between validation and test results might be due to a property of the test set not considered when building the validation set: the users who contributed for the photos in the training set are di erent from those who contributed for the photos in the test set.
Regarding the distribution of test results, for the visual descriptors (runs 2 and 3), the 1st Quartile shows that 25% of the items were geocoded at most 1,900km from the correct location. On the other hand, for the textual descriptors and their combinations (runs 1, 4, and 5), 25% of the items are very close to their correct locations (less than 3km). 4.
Our best results were observed for the methods based only on textual description. For them, we could geocode within 1km radius around 20% of the testing set (test3). Considering visual descriptors, the main challenge this year was the large scale dataset, which poses time and space constraints in the descriptors to be used. Our rank aggregation strategy, for the test set, was only e ective for combining textual descriptors. Combining textual and visual descriptors did not improve the results. As future work, we would like to evaluate a more elaborate geocoding approach, similar to the scheme used to create our validation set, for example.
We thank the support of FAPESP (2011/11171-5, 2009/10554-8), CNPq (306580/2012-8, 484254/2012-0), CAPES, FAPEMIG, Samsung, ACM SIGIR, and MediaEval organizers.