In this paper, we describe our approach as part of the MediaEval 2013 Placing Task evaluation. We use language model and similarity search as baseline approach, and improve the accuracy by two techniques: photo set refinement and tag segmentation. The first technique takes advantage of geo-location correlation among test photos and the second one exploits the textual similarity between tags.
A total of 8,539,050 geo-referenced photos from Flickr were
provided as training data. Following [
The framework proposed by [
Photos within the same set or collection would be highly geolocation correlated. For example, a user can upload his photos which were taken on during a trip into a new set created by him. However, not every photo in the same set is well tagged because a user only tags the photos he loved or interested in, and leaving others un-tagged or poorly tagged. This will result in photos with completely different tag sets or visual content could be considered as taken in the same location or nearby, if they were within the same photo set.
A test photo with poor tags will result in a bad estimation. However, if this photo belongs to a photo set which contains one or more photos with well estimated location (usually well tagged), then we can use the centroid location of these photos as the estimation for the bad one. This is the intuition of our proposed photo set refinement, and there are two problems here: 1. Given a photo, how to find its neighbors within the same photo sets? 2. How to distinguish between the well estimated photo and bad one? Although we didn’t handle the Placeability sub-task of Placing Task at MediaEval 2013, our solution for the second problem may be considered as a naive approach for error estimation.
To handle the first problem, it seems we can simply break down the test data into different sets according to the original photo sets created by users. However, a photo set in this user scenario can be 1 http://www.flickr.com/help/collections/ 2 https://developers.google.com/maps/documentation/geocoding/ run2 run4 changed from time to time, whether it’s adding new photos or deleting the old ones. And the geo-location correlation between these photos will become weaker. Therefore we need a different approach: Given a photo, we find its neighbors in the test data by comparing their user id, the timestamp of the photo was taken on and uploaded. If a photo has an identical user id with the given photo, and the time interval between their taken dates is less than , and their uploaded dates interval is less than
, then we consider these two photos belong to the same photo set. Here both thresholds ( and ) are set to 7 days because we consider a week-long vacation is common for most people, and photos taken and uploaded during these days can be consider as a photo set.
There are three clusterings of the training data, namely , and , and a given test photo can be classified to three different medoids respectively, which we referred to as and . Intuitively, these three medoids are not far from each other if is well estimated and vice versa. So given a photo set , we consider as well estimated if all the Haversine distances among , and are less than 1000km, otherwise is marked as badly estimated. Finally, we use the centroid location of well estimated photos as the finial estimation for the poorly estimated ones, and if no well estimated photo is found, we use the home location of the user (in run 5 only) or simply leave it unchanged. ,
Consider the tag = ‘southchinauniversityoftechnology’ and
tag = ‘southchinauniversityoftechnologylibrary’. If was an
unseen tag, it will be ignored even though we can assume that
and are correlated because of their textual similarity. However,
we can split into two terms ‘southchinauniversityoftechnology’
and ‘library’, then the first term is identical to and can be used
for georeferencing. Our approach for tag segmentation is to model
the distribution of the segmentation output. First, we assume all
tags are independently distributed, and the relative frequency of
all tags in the training data was calculated. We created a tag
dictionary sorted in descending order with size 2,080,618. We
also assume that the tags in the training data follow Zipf's law [
Given a test photo, all the tags in this photo are preprocessed by tag segmentation before georeferencing. For each tag, we select its longest split term and assign it to this photo as a new tag. The remaining terms (such as ‘library’) are discarded because these terms are usually not spatially relevant.
We submitted five runs and the results of our experiments are shown in Table 1. run1: is the baseline approach run2: uses visual features only and K-nearest neighbor search. run3: corrects poorly estimated photos in run1 by photo set refinement proposed in section 2.3. 66.3 0.6 69.2 69.6 76.5
ME km 37.65831 10026.17 28.01581 27.0791 16.73021 run4: is similar to run3 but tag segmentation is used to preprocess the test data before georeferencing. run5: uses the user home location in the photo set refinement step.
Note that this location is also used when estimating the prior
probability in language model framework, we refer to [
The result of run3 justifies our assumption and we can estimate test photos jointly to improve the accuracy. In our experiment, the number of different estimated photos between run1 and run3 is 4,963, and this is the number of photos changed during the photo set refinement step. After comparing the georeferencing result of run1 and run3 with the ground truth, among these 4,963 photos, we found that 4,390 photos’ estimated location in run3 became closer to the real location in comparison with run1, and the rest of 573 photos had a larger error distance in run3 compared with run1. This is mainly caused by the incorrectness of differentiating well estimated photo and the bad one. For some well estimated photos, the Haversine distances among their , and could be far from each other. Therefore, we need a much more robust way to find out the error estimation.
Run4 doesn’t show a promising improvement compared with run3. The reason is that unseen tags are not always segmentable, but the proposed technique did improve the performance slightly and the extra time and computational costs are low. However, other than tag segmentation which only exploits the textual similarity between unseen tags and training tags, we can also try to find out the semantic similarity between them by utilizing external resource or machine learning technique.
Run5 indicates that the home location of the user is very important for georeferencing for most photos, which is consistent with previous research findings. In run2, we simply used the extracted visual features provided by task organizers and ran a Knearest neighbor search to find the most similar photo in the training set. However, we didn’t get a reasonably geo-location prediction and more intensive study is needed in our future work.