We present the results of a system that georeferences Flickr videos using a combination of language models and similarity search. The system extends our approach from last year by using language models with a more adaptive granularity, and by taking into account the home location of the user. Postdoctoral Fellow of the Research Foundation { Flanders (FWO).
The Placing Task requires participants to estimate the
geographical coordinates of a video, based on the visual and
auditory features of the video, textual tags that have been
assigned to it by its owner, context information about the
owner, etc. Training data consists of a portion of the
georeferenced photos on Flickr. For a detailed description of this
task, we refer to [
We participated in the 2010 Placing Task with a system
based on a two-step approach [
A total number of 3 185 258 georeferenced photos from Flickr were provided as training data by the task organizers. As last year, photos that have been uploaded on the
Y t2tagsk(x)
1 P (tja)A
P (a)
where tagsk(x) is the set of tags from Vk that have been
assigned to video x. The probability P (tja) is estimated
using Bayesian smoothing (see [
P (a) /
1 d(pa; phome) where d refers to geodesic distance, pa are the coordinates of the most central photo of area a (i.e. the medoid of the locations of the photos from the training data located in area a) and phome are the coordinates obtained from the textual home location using the Google Geocoding API1. The parameter was set to 0.75 in our experiments. If coordinates of the home location cannot be obtained, P (a) is estimated as the percentage of all photos from the training 1http://code.google.com/apis/maps/documentation/ geocoding/ (1) (2) data that are contained in area a, i.e. (3) (4) P (a) = P
jaj a2Ck jaj identifying a with the set of photos from area a in the training data. In run 1, where a textual home location may be available, but gazetteer look-up is not allowed, (3) can be rened by looking at tags from the vocabulary Vk that appear in it:
Y t2homeTags(x)\tagsk(x)
jaj a2Ck jaj where
was set to 0.45 in the experiments.
The language modeling approach to georeferencing
requires an appropriate level of granularity to be determined:
for videos with more informative tags, it is bene cial to
consider a ner-grained clustering. As a baseline technique for
selecting the optimal value of k, we check the number of
tags a video x has in common with the di erent
vocabularies. If tags7500(x) \ V7500 t7500, with t7500 an
appropriate threshold value, k = 7500 is chosen. Otherwise, if
tags5000(x) \ V5000 t5000 we select k = 5000, etc. For run
1 and 2 the threshold values where chosen as t500 = 1 and
t2500 = t5000 = t7500 = 2. For run 3, on the other hand,
we set t500 = t2500 = t5000 = t7500 = 1. Run 4 is not
based on language models. For run 5, we used a technique
based on Dempster-Shafer theory which was proposed in [
Once a suitable value of k has been chosen, the area a from Ck that maximizes (1) is determined. Subsequently the photo from area a (in the training data) which is most similar to the video x is determined, and its location is used as the prediction for the location of x. Similarity is determined by comparing the tags assigned to each photo with the tags assigned to x using Jaccard similarity (without feature selection).
As a fall-back strategy, if no tags have been assigned to
x at all, the home location of x is used as the prediction
(in those runs where the use of a gazetteer is allowed). If
no home location is available, we use the location of the
photo which is visually most similar to video x. To
measure visual similarity, a photo is compared against the key
2Speci cally, the most likely area was determined using the
pignistic probability decision rule [
The results of the ve runs are provided in Table 1. In particular, the table shows how many of the 5347 videos in the test collection were localized within 1km, 10km, 100km, 1000km and 10000km of the correct location.
As can been concluded by comparing the results of runs 1 and 2, using the geocoded home location is really boosting the results. Also, determining a good threshold value to fall back to a coarser clustering can impact the results, as is demonstrated in run 3 which only di ers from run 2 in its choice of the threshold values t500, t2500, t5000 and t7500. Run 4 is a baseline run which only uses visual features. Unsurprisingly, run 5, which is based on a larger training set, yielded the best results. As further experiments have indicated, however, this increased performance is not only due to the larger training set, but also to the use of DempsterShafer theory to combine the di erent granularity levels.