The goal of the task is to produce location estimates for a test set of 500,000 images and videos using a set of approximately five million geotagged images and videos and their metadata for training. A full description of the challenge and of the associated dataset is provided in [1]. Our runs were implemented using, for a large part, methods described in CEA LIST's participation at Placing Task 2013 [2]. For this reason, after a short presentation of the methods, runs and obtained results, we focus on failure analysis.
Language models are successfully introduced in [3] as an alternative to gazetteer-based geolocation and were progressively improved in following years. Test photos can be placed anywhere in the physical world and the training data provided by the organizers is insufficient in order to build robust probabilistic models. To verify the assumption that better results are obtained with the use of more data, we exploited:
(1) all geotagged metadata from the YFCC dataset1 , after removing all test items and (2) an additional set of ∼90 million geotagged metadata from Flickr.
Similar to last year [2], the surface of the earth was split in (nearly) rectangular cells of size 0.01 of latitude and longitude degree (approximately 1km 2 size). User counts were used instead of tag counts in order to mitigate the influence of bulk tagging. Both titles and tags were taken into account and are referred to as tags hereafter. Put simply, we
Copyright is held by the author/owner(s).
MediaEval 2014 Workshop, October 16-17, 2014, Barcelona, Spain computed the probability of a tag in a cell by dividing its user count in that cell by its total user counts in all cells. Given a test item, we simply summed-up contributions of individual tags to find the most probable cell for that item. Finally, the photo was placed at the center of the cell.
The authors of [4] show that machine tags can improve automatic geotagging quality. In [2] we propose a machine tag processing method which models only machine tags which are strongly associated to locations. (i.e. Foursquare, Lastfm and Upcoming entries) and we exploited it this year.
If images do not have associated tags or if these tags are not geographically discriminant, placing photos with probabilistic models is likely to fail. To overcome this problem, we exploited a simple user modeling technique [2], which computes the most probable cell of a user. Only photos which are at least 24 hours away from any of the user's test set images were exploited to reduce the risk of learning from test data. We downloaded up to 500 geotagged images per user in order to determine her most probable cell.
We propose a late fusion scheme which is empirically derived from tests with a validation dataset. Since they are associated to precise locations or geolocated events, processed machine tags are very reliable and were used in priority. If there were no machine tags, location models were exploited to predict the most probable location of a set of tags. Finally, if there were no tags available or if the prediction score was below a threshold, the photo was placed in the most probable cell of the user who uploaded it. The threshold for replacing location models with user models was empirically determined on the validation dataset. We exploited user models for the 30% of test images which had the lowest placing scores.
We submitted the following runs: RU N1 -exploited location models and machine tags from training data provided by the organizers; RU N3 -combined location models and machine tags from the entire geotagged YFCC dataset, after excluding test items; RU N4 exploited tags and user models; RU N5 -exploited YFCC location models, machine tags and user models. We present the performance of the submitted runs in Table 1. The best results were obtained when com-P@X km Run 0.01 0. bining all types of available information. As expected, the largest contribution was due to location models. The large gap between RU N1 and the others confirms that the use of supplementary training data is very beneficial. The difference of precision at close range (P@0.1) between RU N3 and RU N4 confirms that machine tags are very useful for precise geolocation. Inversely, if larger errors are admitted, user models become more useful than machine tags. The combination of these types of cues in RU N5 gives the best performance for all precision ranges. The results obtained this year are in the same range as those we reported in 2013 [2], confirming thus that our geolocation pipeline has consistent behavior over different datasets.
In addition to the submitted runs, we tested other configurations which gave lower results and briefly describe them here. We notably tried a combination of location models and gazetteer information in order to give a privileged role to toponyms such as administrative division names (i.e. countries, regions, cities). The addition of the gazetteer gave lower results compared to the sole use of location models. This negative result could be explained by the strong ambiguity which characterizes the geographic domain. As we mentioned, we also tried to add a dataset of ∼90 million geotagged metadata to the YFCC full training data. Contrarily to existing literature [4,2], the use of this supplementary dataset actually degraded the overall quality of results. This negative result might indicate that probabilistic models reach saturation when too much metadata are available.
In Figures 1 and 2, we present a visualization of geotagging performance for RU N1 and RU N5 and the performance difference between the two runs is clearly reflected . Geotagging is precise in most European regions and worse for the other regions. Low performances can be easily explained by sparse data for Africa, Asia or South America. However, the imprecision is also high for the United States, the region of the world which concentrates the largest number of geotagged images. In this case, poor geotagging could be due to a very high ambiguity of place names. For instance, there are dozens of places called London or Paris in the US. If there is not enough disambiguation information associated to them in annotations, photos tagged with these toponyms will be placed in Europe.
Due to lack of time, we did not submit a visual run this year. While visual geotagging still lags well behind textual geotagging, it would be interesting to explore if it is possible to predict coordinates accurately at least for visually distinctive objects such as Points of Interest. Regarding text models, we would like to investigate in more depth why adding more data from outside YFCC degrades performance. Equally interesting, it would be interesting to investigate ways to select reliable annotations before computing location models.