RECOD Working Notes for Placing Task MediaEval 2011 ∗ Lin Tzy Li, Jurandy Almeida, and Ricardo da S. Torres Institute of Computing, University of Campinas – UNICAMP 13083-852, Campinas, SP – Brazil {lintzyli, jurandy.almeida, rtorres}@ic.unicamp.br ABSTRACT <"+1'/-0*@B 8'*(0('$"-%4'$B4+*@)% !"#$% "70("*."-)%-.+/"% This work is developed in the context of placing task at <"+.+(0*@% MediaEval 2011. It consists in automatically assigning geo- &"$'('$')%'**+$',+*-)% graphical coordinates to a set of videos. Our group proposed ("-./01,+*-% <"+%+*$+4+@F% an architecture design for the multimodal geocoding. In this <"+% !6"-'3/0% paper, we focused on implementing a simple content-based 20-3'4%5"'$3/"-% =">'*,.% <'G"H""/-% <:;% ?9% approach, which is part of the proposed framework. The Data fusion reported results show our strategy compared to those from 89:;% 8+>E0*"% previous year participant using only visual content to ac- 20("+% &'$.6% 8'*(0('$"-%4'$B4+*@)% @"+.+(0*@% "70("*."-)%-.+/"% complish this task. 1/+."--0*@% 70("+% /"-34$-% Categories and Subject Descriptors 8'*(0('$"-%4'$B4+*@)% H.3.3 [Information Search and Retrieval] <"+.+("(% "70("*."-)%-.+/"% 8'*(0('$"-%4'$B4+*@)%% 20("+% 20("+%A0$6%4'$B4+*@)%% $'@-)%('$")%("-./01,+*% $'@-)%('$")%("-./01,+*% C%>'$.6%-.+/"% C%.+*D("*."%-.+/"% 1. INTRODUCTION The geographic information is present in people’s daily life, thus it is not surprising that there is a huge amount Figure 1: Multimodal geocoding proposal of data on the Web about geographical entities and a great interest in localizing them on maps. That information is 2. THE PROPOSED FRAMEWORK often enclosed in digital objects (e.g., documents, image, The proposed architecture for dealing with multimodal and videos). Once they are geocoded (i.e., associated to a geocoding is composed by three modules (Figure 1): (1) latitude or longitude), one can perform geographical queries. text-based geocoding; (2) content-based geocoding; and (3) Current solutions for geocoding multimedia material are data fusion/rank aggregation-based geocoding. The first usually based on textual information [2, 6]. Such a strategy module is in charge of geocoding based solely on textual depends on the human intervention to tag textual descrip- part of the digital object. Content-based geocoding module tions of the data. However, there is a lack of objectivity and is responsible for dealing with and geocoding based on its completeness of those descriptions, since the understanding visual content. Finally, the rank aggregation-based module of the visual content of multimedia data may change ac- combines the results generated by the previous modules and cording to the experience, and perception of each subject, gives the final result of the geocoding. The idea is to rely not to mention lexical and geographical problems in recog- on text and image whenever possible. nizing place names [5]. This opens new venues for the in- In this paper, we focused on the second module, exploring vestigation of methods that use image/video content in the a method to identify similar videos whose visual content geocoding process. Furthermore, data fusion/rank aggrega- indicates where those videos were filmed. Although it is tion approaches could be also used for combining evidences allowed to use all the metadata associated to the given video, found in both textual and visual content. such as descriptions and tags provided by users, we focused In this paper, we present an approach for visual content- on geocoding based on visual features of the videos. based geocoding, although we aim to explore the combina- tion of textual and visual content of digital objects in order to improve their geocoding. The idea here is to test how 2.1 Extracting & Comparing Visual Features well video similarity in term of its motion sequence would Instead of using any keyframe visual features provided by fit our purposes of predicting their location. the organizers, we adopted a simple and fast algorithm to This work is developed in the context of Placing Task at compare video sequences described in [1]. It consists of three MediaEval 2011. The goal of such a task is to automatically main steps: (1) partial decoding; (2) feature extraction; and assign geographical coordinates (latitude and longitude) to (3) signature generation. a set of annotated videos. More details regarding data, task, For each frame of an input video, motion features are ex- and evaluation are described in [7]. tracted from the video stream. For that, 2×2 ordinal matri- ces are obtained by ranking the intensity values of the four ∗ We thank FAPESP, CNPq, and CAPES for financial support. luminance (Y) blocks of each macroblock. This strategy is employed for computing both the spatial feature of the 4-blocks of a macroblock and the temporal feature of cor- Copyright is held by the author/owner(s). responding blocks in three frames (previous, current, and MediaEval 2011 Workshop, September 1-2, 2011, Pisa, Italy next). Each possible combination of the ordinal measures Table 1: Results using only videos visual content (distance between ground truth and estimated) Radius (km) 1 10 20 50 100 200 500 1000 2000 5000 10000 Dev set: % in range 14.42 16.02 16.44 16.93 17.51 18.36 21.20 26.04 34.76 46.76 84.29 Test set: % in range 0.21 1.12 1.59 1.93 2.71 3.33 6.08 12.16 22.11 37.78 79.45 is treated as an individual pattern of 16-bits (i.e., 2-bits for determined 317 classes for the SVM with the descriptors each element of the ordinal matrices). Finally, the spatio- CED, FCTH, and Gabor. They presented their results for temporal pattern of all the macroblocks of the video se- video’s location correctly predicted within radius of 50 km, quence are accumulated to form a normalized histogram. 100 km, 200 km, 750 km, and 2,500 km. For a detailed discussion of this procedure, refer to [1]. In order to compare our results to those presented by Kelm The comparison of histograms can be performed by any et al. [3], we aggregated the evaluation results presented vectorial distance function like Manhattan (L1 ) or Euclidean in Table 1 according to their experimental protocol. This (L2 ) distances. In this work, we compare video sequences regrouping was possible due to the placing task organizers, by using the histogram intersection, which is defined as who made available to all participants of that task: their P i i tool to calculate the distance (Haversine distance formula) i min(HV1 , HV2 ) from ground truth to the estimated location for each result; d(HV1 , HV2 ) = P i , i HV 1 and the test videos ground truth. Table 2 compares our approach with the results reported where HV1 and HV2 are the histograms extracted from the by Kelm et al. (adopted from their Table 6) [3]. Notice that videos V1 and V2 , respectively. This function returns a real our method, although simpler, shows high precision rela- value ranging from 0 for situations in which those histograms tive to their clustering-and-classification method. The key are not similar at all, to 1 when they are identical. advantage of our technique is its computational efficiency. 2.2 Geocoding the Visual Content Unlike them, we did not use any data to train any classifier. We used 10,216 videos from the development set released Table 2: Our regrouped test results vs. Kelm et al. by Placing Task organizer as geo-profiles against which each Radius (km) 50 100 200 750 2500 test video was compared to. Our approach % 1.93 2.71 3.33 9.18 24.48 In order to assess how well we did, only relying on visual Kelm et al. % 3.38 5.26 6.23 10.65 19.92 content during the development phase, we extracted the vi- sual content of each provided video, then we compared all 4. CONCLUSIONS videos of the set against each other, and finally, for each Relying just on video content to estimate its location still video, we produced a list of videos ordered by similarity in poses a challenge. It seems that this task requires using descending order. Considering that a query video always is textual information found in video metadata such as de- the best match to itself, thus it will be the first in this list, scriptions, user tags, external knowledges bases as shown by we took the second video from the top list as the one that some related works. will transfer its known lat/long to the query video. Our method used the video similarity between videos in For the test result, applying the visual feature extraction development set and those in test set to estimate location of and the similarity computation explained previously, each those. The similarity in this work is given by motion pat- video in test set (5,347) was compared with those in the terns extracted from the video streams. This algorithm is development set. Then, for each test video, an ordered list of simple and achieved comparable results to those more com- similar videos from the development set was produced along plex presented by previous work that also was based just on with its similarity score to that given test video. Finally, video visual clues. we picked the most similar video of this list as the one that We believe that we can improve the results by developing will transfer its known lat/long to the query test video, and new video similarities approaches as well as new combining reported that lat/long as the one to be given to test video. methods for image and textual evidences in the context of geocoding digital objects. 3. EXPERIMENTAL RESULTS For this task, we performed one submission for the run that considered just visual content. The evaluation results 5. REFERENCES [1] J. Almeida, N. J. Leite, and R. S. Torres. Comparison of are shown in Table 1. Note that, by relying just on video video sequences with histograms of motion patterns. In similarity based on its visual content, our algorithm will hit ICIP, 2011. 79.45% only when accepting an error of 10,000 km between [2] C. B. Jones and R. S. Purves. Geographical information the ground truth and the assigned point. However, when retrieval. Int. J. Geogr. Inf. Sci., 22(3):219–228, 2008. considering 100 km of error, it predicts lat/long correctly [3] P. Kelm, S. Schmiedeke, and T. Sikora. Multi-modal, for only 2.71%. These results underperform those from the Multi-resource Methods for Placing Flickr Videos on the reference algorithm for this task (winner of the last year), Map. In ICMR, pages 52:1–52:8, 2011. which just analyzes user-contributed tags for predicting the [4] M. Larson, M. Soleymani, P. Serdyukov, S. Rudinac, et al. Automatic tagging and geotagging in video collections and geotag of a video (73.6% of videos are within 100 km) [4]. communities. In ICMR, pages 51:1–51:8, 2011. However, we are interested in comparing to other results [5] R. R. Larson. Geographic information retrieval and digital using only video content to accomplish the placing task. For libraries. In ECDL, volume 5714/2009, pages 461–464, 2009. instance, Kelm et al. [3], who also reported their results [6] J. Luo, D. Joshi, J. Yu, and A. Gallagher. Geotagging in when only visual content of test videos were used to predict multimedia and computer vision–a survey. Multimedia Tools their location on Earth, have used visual features of the de- Appl., 51(1):187–211, 2011. velopment set for training a multi-class SVM classifier with [7] A. Rae, V. Murdock, P. Serdyukov, and P. Kelm. Working RBF kernel. Their best results were achieved by a hierarchi- Notes for the Placing Task at MediaEval 2011. In MediaEval, 2011. cal clustering with a diameter threshold of 100 km, which