This paper describes Imcube's geo-referencing approach, experiments, and results at the MediaEval 2015 Placing Task benchmark. This task requires to develop techniques to automatically annotate Flickr photos and videos with their geolocation (latitude and longitude) in two individual subtasks. A hierarchical approach combining textual, visual and optional routing information is employed. The results show that for 24% of the images (local-based task) and for 96% of the images (mobile-based task) the error of the estimated location is below city level (10 km).
The MediaEval Placing Task 2015 [
This year the task introduces two new sub-tasks: The localebased sub-task addresses the prediction of missing locations for individual images in an entity-centred way by choosing a location from a given ground truth hierarchy. The mobile-based sub-task addresses predicting missing locations within a sequence of photos shot by a travelling photographer.
Similar to the Placing Task 2014, the training set (4,672,382
photos & 22,767 videos) and test sets (931,573 photos & 18,316 videos)
were sampled from the YFCC100M [
In this paper, we present an approach that combines different textual and visual descriptors by applying a hierarchical scheme to merge information obtained from several ranked lists.
This section describes the different methods created to solve the challenges of the locale-based and mobile-based sub-tasks.
The proposed approach is composed of different steps: (i)
hierarchical clustering of the provided training set by latitude and
longitude, (ii) visual and textual feature extraction, (iii) generation
of ranked lists, (iv) re-ranking and (v) estimation of the location for
each test item.
The hierarchy provided contains 221,458 leaf nodes (locations)
that are spread across 253 countries. Below the second level
(Country State) we segment the states into 360 180 regions
according to the meridians and parallels. We also apply a smaller grid
of segments with half the spatial dimensions to increase the
accuracy and to minimize the computational cost. Each geo-referenced
training image is assigned to its corresponding grid cell at the
lowest level [
Due to the large size of the dataset and the limited processing
time, we did not apply a hierarchical language model approach with
multiple modalities [
The visual similarity relies on a wide spectrum of visual features to describe the color and texture characteristics of the video key frames and photos. These image descriptions are pooled for each leaf node in the different hierarchy level using the mean and median value of each descriptor. A kd-tree that contains all appropriate segments is built for each descriptor in each leaf node. This procedure speeds up the following search because only a portion of data is needed to be computed for nearest neighbour search.
Starting at the top of the hierarchy the nodes of the current level are ranked according to their distance to the test image. The overall distance is obtained by fusing the textual and the visual distances using weighted summation. These weights differ in both fusion experiments as described in results section. Then the node with the lowest distance becomes the most likely location at the given level of granularity. By iteratively traversing the hierarchy the method determines the leaf node that has the highest similarity to the test image and returns the corresponding geolocation. 2.2
For this task, we pursue a similar approach as described in section 2.1 but without the hierarchical layer model and with additional routing information.
We use OpenStreetMap3 to find the shortest route between two photos that have associated geographic coordinates. Tracks with a distance smaller than 2 km are routed by pedestrian navigation, larger tracks are routed by car navigation, respectively. The results 1http://tartarus.org/martin/PorterStemmer/ index.html 2http://lucene.apache.org/core/ 3http://www.openstreetmap.org/ #Items 2 81 1593 6501 9171 9659 9674 9675
Routing
Percentage 0.02 % 0.84 % 16.47 % 67.19 % 94.79 % 99.83 % 99.99 % 100.00 % of the routing run are predicted linearly in travel time to be a location on these tracks. For test images, which do not have both chronological neighbours, the neighbouring route segment is extrapolating while considering their distance in time. The other runs use additionally textual and visual features to determine the most similar image along the track.
The visual similarity is determined as described in [
The textual run uses the same textual similarity as the location task, but the training images are restricted to be located within a corridor of 0.001 degree along the estimated routes.
Table 1 shows the accuracies of selected error margins for the different textual and visual runs. Based on the experience from the previous years we expect the textual run to perform better than the visual run due to the visual ambiguity at coarser levels. The results clearly show that the visual only approach has low accuracy in all error margins when compared to the textual only approach. For combining the textual and visual information we have tested two different fusion models. We design a set of two fusion experiments to combine textual and visual features. Our first fusion model (Fusion1) combines the estimations of textual and visual models equally on each hierarchy level. The second fusion model (Fusion2) only combines these estimations on the finest three hierarchy levels. On the coarsest hierarchy levels, only the estimation of the textual model is used. This combination results in more accurate results, since visual feature are not able to solve ambiguities in large scale (i.e. , most cityscapes look similar). The results show that fusing visual and textual information on finer levels (Fusion1) improves the performance for error margins between range between 10 m and 100 km. 3.2
The results of the local-based sub-task show that the best performance can be achieved with a multimodal fusion approach that uses textual information on coarser levels and the combination of visual and textual information in finer ones. The results of the mobilebased sub-task show that the use of visual and textual information beside routing information improves the location estimation. The low correlation of the localization errors of the different approaches suggests that more advanced fusion approaches will lead to better results. Another interesting direction to improve the accuracy of the visual approach for both sub-tasks is by using local features to distinct landmarks and points of interest.