We describe the participation of the SocialSensor team in the Placing Task of MediaEval 2014. We submitted three runs based on tag information for the full test set, using extensions over an existing language modelling approach, and two runs (one based on the full test set and the other on the 25,500 subset) based on visual content, using geospatial clustering and supervised-learning. Our best performance (median error 230km, 23% at 1km) was achieved with the use of tag features, using only internal training data.
The goal of the task is to produce location estimates for
a set of 510K images using a set of over 5M geotagged
images and their metadata for training [
Baseline approach: The baseline method relies on an ofine step, in which a complex geographical-tag model is built from the tags and locations of the approximately 5M images of the training set. The metadata used to build the model and the estimation of a query image are the tags, the title and the description. A pre-processing step was rst applied to remove all punctuation and symbols and to transform all characters to lower case. After the pre-processing, all training images left with empty tags and title are removed, resulting in a training set of approximately 4.1M images. Note that the same pre-processing is applied on the test images before the actual location estimation process.
In contrast to last year's clustering [
In order to assign a query image in a cell, we calculate the probability of each cell summing up the contributions of individual tags and title words. The cell with the greatest probability is selected as the image cell. If during this process there is no outcome (i.e. the probability for all cells is zero), we use the description of the query image. For the test images where there is no result (e.g. complete lack of text), we set their location equal to the center of the most populated cell, of a coarse granularity grid (100km 100km), a kind of maximum likelihood estimation.
Extensions: We devised the following extensions:
Similarity Search: Having assigned a query image to a cell,
we then employ the location estimation technique of [
Internal Grid: In order to ensure more reliable prediction in ner granularities, we built the language model using a ner grid (cell side length of 0.001 for both latitude and longitude, corresponding to a square of 100m 100m). Having computed the result from both the coarse and ne granularity, we use an internal grid technique. According to this, for a query image, if the estimate based on the ner granularity falls within the borders of the estimated cell of the coarser granularity, then we consider the ne granularity trustworthy and apply similarity search inside the ne cell. Otherwise, we perform similarity search inside the coarser granularity cell, since coarser granularity language models are by default more trustworthy (due to the use of more data for building them).
Spatial Entropy: In order to adjust the original language model tag probabilities for each cell, we built a Gaussian weight function based on the values of the spatial tag entropy. The spatial entropy for each tag tk is calculated based on its probabilities over all m cells of the grid.
e(tk) = m X p(tkjci) log p(tkjci) i=1 We chose a Gaussian model because the tags with either too high or too low entropy values typically carry no geographic cues, and we would therefore need to suppress their in uence on the location estimation process. Equation 2 presents the entropy-based cell estimation equation.
T p(cijj) = X P (tkjci) k=1
N (e(tk); ; ) (2) where p(cijj) is the probability of cell ci for image j, T is the number of tags for image j, P (tkjci) is the probability of tag k for cell ci and ek is the value of the entropy of tag k. N is the Gaussian function, and the parameters ; are estimated using the distribution over the training set. 2.2
To build the visual location models, we relied on two
features, SURF+VLAD and CS-LBP+VLAD, concatenating
them in a single vector. In particular, we rst calculated
the interest points of each image, and then extracted both
SURF and CS-LBP descriptors corresponding to them. The
parameters used for CS-LBP [
The main part of the model building included the training of linear SVM to separate the samples in a prede ned number of spatial clusters and subclusters (we used 50 clusters and up to 50 subclusters corresponding to each cluster). The clusters/subclusters were created using k-means on the coordinates of the training images, while the number of subclusters was determined by the number of samples N assigned to each cluster (min(round(N=3000); 50)). Subcluster Selection: For each cluster a one-vs-rest approach was applied resulting in 50-d prediction score vectors, while for the subclusters a similar approach was used but only for intra-cluster samples (resulting in better performance than using both intra- and inter-cluster samples). The decision about the cluster membership for each sample was a combination of the estimation scores provided by the cluster prediction score vectors and also the scores corresponding to the best subcluster score in each cluster. Finally, priors (based on number of images per cluster/subcluster) were applied to the respective scores, since they were found to lead to some improvement.
Similarity Search: To achieve location estimations of ner granularity, we applied a similarity search step at the subcluster level. In particular, the query image was compared to 1000 samples from the selected subcluster (sampling was necessary for e ciency reasons), and the location of the most similar of those was returned. Similarity was computed based on a low-dimensional concept-based representation, using the 94 concepts of ImageCLEF 2012 (i.e. each image was represented by 94 prediction scores coming from a set of corresponding pre-trained concept models). measure acc(10m) acc(100m) acc(1km) acc(10km) acc(100km) acc(1000km) median error
As described above, we prepared three tag-based runs and two visual runs. The tag-based runs are the Run 1, using the language model, similarity search, internal grid and spatial entropy, Run 4, using the language model and the center of cells as estimated location, and Run 5, using the language model and similarity search. Run 2 was based on the Subcluster Selection step of subsection 2.2 using the center of the subcluster as location estimate. Run 3 was based on the combination of Subcluster Selection with Similarity Search (according to subsection 2.2). For Run 3, we used a subset of 25,500 images due to lack of time. For the rest of the runs we used the full test set of 510K images.
According to Table 1, the best performance in terms of both median error and accuracy in all ranges was attained by Run 1. Comparing Run 4 and 5, it can be seen that similarity search had considerable impact on the low range accuracy results. Also the combination of all features in Run 1 improves further the overall performance (reaching a 5.85% accuracy for the < 100m range), but the median error is still quite high (230km), which means further improvements can be achieved. The visual runs yielded very poor results.
In the future, we plan to look into utilizing external data for training, in particular the Flickr 100M Creative Commons dataset and gazetteers. Furthermore, we will look into alternative ways to utilize visual information for geotagging. Acknowledgements: This work is supported by the SocialSensor FP7 project, partially funded by the EC under contract number 287975.