This paper describes our contributions to the Social Event Detection (SED) task as part of the MediaEval Benchmark 2014. We rst present an unsupervised approach for the clustering of social events that builds solely on provided metadata. Results show that already the use of available time and location information achieves high clustering precision. In the next step, we focus on the retrieval of previously clustered social events from queries by using temporal, spatial, and textual cues.
The immense daily growth of publicly available photos
introduces the need for approaches that are able to e
ciently mine large photo collections. A signi cant part of
shared content depicts a variety of di erent social event
types. Hence, a lot of recent research focuses on the
detection, classi cation, and retrieval of social events. The Social
Event Detection (SED) task of the MediaEval Benchmark
provides a platform for the development and comparison of
such approaches [
In 2014 we participated in subtasks 1 and 2 of the SED
task [
We propose an unsupervised, three-stage approach for the clustering of images into social events. Initially, each image is assigned to a single item cluster. At each stage we perform re nement and merging of previously detected events by considering a di erent aspect of the available image information, ranging from user and capture time information via location data to user-provided textual descriptions.
In the rst stage, temporal-based clustering, we employ an adaptive approach to merge the initial single item clusters. Since a user can only be present at a single event within a prede ned time span, we explore the time di erence between consecutive images captured by the same user. If it is within a prede ned threshold, the corresponding images are assigned to the same event cluster. In the next stage, we apply the same adaptive approach for location-based clustering. If the minimum time and location distances between two event clusters are within the prede ned thresholds, they are merged. As a result, detected events can vary strongly in both their duration and size. A di erent approach for location-based clustering is using a prede ned xed radius for the identi cation of social events. For every event cluster resulting from the rst stage a representative location is estimated by calculating the sum of distances from each geo-tagged photo to all other geo-tagged photos in that cluster. The location of the photo with the minimum distance to all other photos is the representative location of the event cluster. If the estimated locations of two event clusters are within the prede ned radius, these clusters are merged and the representative location is updated. Event clusters without location information remain unchanged in the second stage of our approach.
The nal stage of our approach is the text and topicbased re nement of previously detected clusters. We extract term dictionaries and topics using Latent Dirichlet Allocation (LDA) from the textual metadata of the images. Temporally and spatially similar clusters with similar textual descriptions are merged by a combined clustering scheme that takes both topic and term similarity into account. Cluster merging and updating is performed iteratively to successively grow clusters. 2.2
For each event cluster we build a TF-IDF representation from the user-generated textual descriptions of the corresponding images. Temporal information is extracted from the metadata provided directly from the photo camera. The location in geo-coordinates of a cluster is mined from available coordinates and from the textual descriptions by using the GeoNames1 database to convert location-speci c strings to geo-coordinates.
As an optional step, additional topic models for the different event types (e.g. music events) of the development queries are generated and a one-class support vector machine (SVM) is trained for each event type. For event retrieval, a global weight (similarity) determines the importance of a given cluster to the query. The global weight accounts for temporal, spatial (city, country, venue), and textual similarity (based on TF-IDF). Additionally, the similarity to a given event type model is considered if one is available for a given test query. Prior to retrieval, the queries are expanded by WordNet2 synsets. All events with an overall weight above 1% of the maximum weight observed for all clusters are returned as result. 3.1
We submitted ve runs for the evaluation of our approach for social event clustering. Runs 1 and 2 are the result of the complete system considering temporal-, location-, and text based clustering. The two runs di er in their location-based clustering only: run 1 is using the adaptive-approach and run 2 the radius-based one. Runs 3 and 4 are the product of the combination of the temporal- and location-based approaches. Eventually, run 5 shows the potential of the use of user and time information only. All runs employ the same parameter settings: time threshold of 24h, location threshold of 1km, and textual similarity of either a term dictionary intersection larger than 0:4 or more than two shared topics.
Table 1 summarizes the results of the evaluation on both the development and test datasets. Achieved results show that the proposed approach generalizes well to the test data. The performance on both datasets is highly competitive given the fact that we only rely on existing metadata. The di erences between runs 1 and 2 and between runs 3 and 4 respectively are negligible and, thus, both location-based approaches deliver robust results for the employed datasets. Noteworthy is run 5 where solely time and user information is considered. The results are only slightly lower at signi cantly lower computational costs in comparison to the text mining stage (runs 1 and 2). 3.2
We submitted three runs. Run 1 is the complete system without query expansion. In run 2 we add query expansion
Development queries
R P F1 Run 1 0.4656 0.8990 0.5367 Run 2 0.5052 0.8974 0.6192 Run 3 0.4770 0.4391 0.3838 and in run 3 we do not use the pre-trained event type models (unsupervised run). Table 2 shows that run 3 yields the highest performance and best generalization ability over all test queries with an average recall of 0:41 and an average precision of 0:42. This is remarkable as this run is completely unsupervised. The performance for the best test query is an F1-score of 74% (query 8). The lowest performance is obtained for the test query 4 (F1-score of 8%). The reason for the di erences in the performance lies in the strongly varying complexity of the queries. Query 8 contains the name of the band "Mogwai" which is highly discriminative and, thus, facilitates the identi cation of relevant clusters. Query 4 asks for "community events" which is highly general (without a more speci c de nition of this category) and, thus, its performance is low. 4.
In this paper we deal with two di erent aspects in the context of social events mining in large media collections. We consider the rst subtask of social events clustering as an unsupervised data mining problem and we additionally refrain from employing any external sources of information. Performed experiments demonstrate the strong generalization ability of the proposed approach and the potential of fundamental metadata such as location and capture time information. The second subtask of social events retrieval indicates the challenge in the mapping between an arbitrary user query and prede ned event clusters. Experiments with optional query expansion and training models show that actually the unsupervised approach that considers available metadata only yields robust performance. The interpretation of abstract queries without any additional information remains an open issue.
This work has been partly funded by the Vienna Science and Technology Fund (WWTF) through project ICT12-010 and the Carinthian Economic Promotion Fund (KWF) under grant KWF-20214/22573/33955.