In this paper, we present a visual-content-based approach that predicts where in the world a social image was taken. We employ a ranking method that assigns a query photo the geo-location of its most likely geo-visual neighbor in the social image collection. The experiments carried out on the MediaEval Placing Task 2013 data set support the conclusion that the exploration of candidate photo's geo-visual neighbors and the combination of local and global image features can improve the prediction accuracy of visual-contentbased geo-location estimation system.
The research question of the Placing Task is how to
estimate the geo-location of one image, given its image
attributes, and all available metadata [
A variety of information sources have been exploited for
predicting geo-location. User-contributed text annotations
have been used as a basis of a large range of successful
geocoordinate predication algorithms [
Our approach consists of four steps, as depicted in the
system overview, Fig. 1. In the first step, Local Feature-based
Image Retrieval, we create a set of candidate photos for
given query q by retrieving all visually similar images based
on local features from the database up to a visual similarity
threshold, k. In the second step, Global Feature-based
Image Selection, we rank all the candidate photos from step one
by their visual similarity with the query based on global
features, and select the top t ranked photos as the final selected
candidate set Evis@k&t. In the third step, Geo-Visual
Ranking, the same as in [
P (qja) / ∑ Simvis(e; q) e2Ea (1) 3. 3.1
where Ea is the set of geographically nearby photos of photo a with high visual similarities to query q. Then, in the final step, Location Propagation, the geo-location of the top ranked photo is propagated to the query photo.
The proposed system is evaluated on a set of 8; 801; 050
geo-tagged Flickr1 photos released by the MediaEval 2013
Placing Task [
Our approach to geo-location prediction exploits visual similarity between photos. To calculate visual similarity based on local features, we choose SURF, and the bag-ofvisual-words scheme to build the search engine. To calculate visual similarity based on global features, we use Joint Composite Descriptor (JCD), which encodes the color, edge directivity, and texture histogram of the image. 1http://www.flickr.com/
RUN 1: Baseline, the system only applies Local Featurebased Image Retrieval, and uses the geo-location of the local feature-based visually most similar photo as estimated location.
RUN 2: Baseline + Geo-Visual Ranking RUN 3: Baseline + Global Feature-based Image Selection RUN 4: Baseline + Global Feature-based Image Selection + Geo-Visual Ranking
The run results are presented in Table 1. In evaluation radius 1km, compared with the baseline method, Run1, Run2 achieves about 18% improvement and Run3 achieves about 17:7% improvement. The best performed one, Run4, achieves about 37:4% improvement, which is more than the sum of the previous two. 4.2
As the query photo is from a social image collection, there are certain properties in the collection that can affect the prediction accuracy of visual-content-based approaches. For example, queries about one popular landmark and queries about an individual user’s car may get different prediction performance. For the purpose of our investigation, we define geo-visual redundancy for a given query photo as the number of photos that are taken within 1km radius of the query photo and also ranked in the top 10; 000 of the rank list of the query from the local feature-based image retrieval system.
The distribution of the query photos over the geo-visual redundancy ranges is illustrated in Fig. 2. Over half of the queries do not have visually similar photos within their geoneighborhood, which suggests that in these cases there is not another photo in the dataset that depicts the same scene or object at the query location. This observation demonstrates how challenging it is to predict the geo-location of a social image purely from its visual content.
Fig. 3 breaks down the geo-location prediction performance over different levels of geo-visual redundancy. Comparing Run3 with Run1, we see that the Global Featurebased Image Selection step can improve the local featurebased system for queries with different geo-visual redundancy level. Comparing Run2 with Run1 and Run4 with Run3, we see that the Geo-Visual Ranking step can boost the performance for queries with high geo-visual redundancy. Comparing Run4 with Run2, we see that the Global Feature-based Image Selection can also benefit the system with Geo-Visual Ranking , especially for queries with medium geo-visual redundancy level.
We have presented a ranking approach addressing the challenging task of predicting geo-location using only the visual content of images. The main observation is that the combination of local and global image features can compensate each other, and together with geo-visual ranking, they improve the prediction accuracy of visual-content-based geolocation estimation system. Future work will include an investigation of the optimal role of local and global features within the geo-visual ranking scheme for visual-content-based geo-location estimation.