There is a wide array of online photographic content that is not geotagged. Algorithms for e cient and accurate geographical estimation of an image are needed to geolocate these photos. This paper presents a general model for using both textual metadata and visual features of photos to automatically place them on a world map.
The primary goal of the 2013 MediaEval placing task [
The motivation for the techniques we have developed for the task was twofold; we rstly wanted to develop a technique that can operate using either the visual content or the metadata, but which also seamlessly allowed blending of information across modalities and allowed information from external gazetteers to be incorporated. Secondly, we wanted our technique to be scalable and e cient, with the aim of being able to estimate the position of an image in well under a second using standard desktop hardware.
The basic idea of our approach is that we estimate a
continuous probability density function (PDF) over the surface
of the Earth from a number of features extracted from the
query image and/or its metadata. To estimate the PDF,
each feature provides a xed size set of points (latitude,
longitude) which are then combined, and a kernel density
estimator can be used to estimate the probability density at
any arbitrary position. By nding the modes of the PDF we
create an estimate of the location of the photograph from
the position of the mode with the highest probability. By
tting a univariate Gaussian over the support of the
highest probability mode, we can estimate the accuracy of the
estimated geolocation as a function of the variance.
In practice, density estimation and mode- nding can be
combined by applying the mean-shift algorithm. Mean-shift
has been used in the context of geolocation estimation in the
past; for example, Hays and Efros [
The implementation of our methodology was realised in
Java using OpenIMAJ1 [
The following features were used in our experiments. Each
feature provides a set of geographic points in response to a
query image:
Location Prior. A constant prior feature built by sampling
1000 geographical coordinates from the training data.
Tags. Every tag in the query image is associated with the
coordinates of the training images in which the tag appeared.
If a tag in the query was unseen in the training data, then
it contributes no points. Each tag is considered to be an
independent feature. No ltering of tags was performed.
PQ-CEDD. In order to provide a high-recall/low-precision
image search we indexed the provided CEDD features with
a product quantiser (18 products of 256 clusters) to enable
fast in-memory search of the complete training data using
the asymmetric distance computation method [
X X X X
X X X
X X X X cations of the 100 top images to a query formed the point set returned by the feature.
LSH-SIFT. High-precision (low recall) image content search
was performed using a variant of the approach we developed
in [
For all of our submitted runs, the methodology described in Section 2 was applied. In all runs we used a at kernel with bandwidth of 0:01 . All runs except for run #5 used a constant number of 1000 points per feature (e ectively making all features have a xed weight); any feature that produced more or less points was randomly and uniformly sub- or super-sampled to reach 1000 points. In total, 5 runs were submitted. The con gurations are summarised in Table 2. Notes about each run are listed below: Runs 1-3: Provided data. The rst three runs used features extracted from the provided dataset only. No external data was used.
Run 4: Text+Visual, bigger dataset. The fourth run used features extracted from a larger dataset of approximately 46 million geotagged Flickr images we collected last year. As with the provided data, only images with an accuracy of 16 were crawled. For the purposes of fair experimentation, we removed all photos from the users who appeared in the test set.
Run 5: Text+Visual, provided data with tag boosting. The fth run was the same as run 1, but we used the GeoNames4 gazetteer to boost the weight of tag features that were likely to belong to a speci c geographic location; any textual tag that could be matched against the the GeoNames \name" or \alternate-name" eld was boosted by doubling its number of points from 1000 to 2000. Non textual features, and tags that didn't match remained at
Our results are shown in Table 1. The rst thing to note is that visual features alone perform relatively poorly for exactly predicting locations; this is be be expected as the vast majority of images do not contain recognisable places. Interestingly our rst and fourth runs (text and visual) performed worse than the third (text only). This was unexpected and warrants further investigation as our experiments had indicated that the visual features should help boost performance. Run 4 indicates that using additional data helps our approach. We'd expect our error estimation to improve (higher correlation coe cient) as we become more accurate; this trend can be seen to an extent, although run 5 is a definite outlier. We had started to experiment with productquantised PCA VLAD encodings of SIFT features, temporal features and query expansion of LSH-SIFT features, however we ran out of time to optimise and include these in the runs. It will be interesting to investigate these further, together with other features such as GIST in the future. It would also be interesting to try some other approaches to incorporating structured knowledge from GeoNames.
The described work was funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreements 270239 (ARCOMEM), and 287863 (TrendMiner).