In this paper, we present our approach and results for the MediaEval 2011 Social Event detection task. VTT participated in Challenge 2 where a given dataset of Flickr photos were matched to events in certain places. We used Linked Data to enhance the dataset by adding event information and other related data and then searching the enhanced dataset. Additional information relating to venues and places were used for creating a subset of photos for each place; Barcelona and Amsterdam. The event profiles including semantically enhanced metadata were used in media retrieval. The approach of combining additional data from the Internet and limiting the queries to limited subsets improved the relevance of photos relating to the events.
Experimentation
The challenges, dataset and evaluation methods of the Social
Event detection task are described in [
In our earlier research [
“profile” for the event and matching it with the metadata of photos.
The main point of the approach was to connect the given photos to events that were found using the Linked Data sources on the Internet. Linked Data was used to get additional information relating to events, artists, venues and places.
First we used publicly available event services such as Last.fm5 and Upcoming6 to find information about the relevant events. The event descriptions including title, description, artist, time and venue information was stored in a database.
By using Freebase we looked up the unique identifiers for the artists and bands. Based on these URIs, additional information such as genre and band members were collected and stored in a database. This additional information was used for updating the “profile” of the event.
We used Freebase and GeoNames for getting additional information relating to places. This included getting coordinates for the venues and cities, as well as different language versions for the cities and countries. We used Freebase for getting information about the tourist attractions in Barcelona and Amsterdam, and GeoNames for getting places near venues by utilizing coordinates. An assumption was that these were things that users commonly use for describing the photos.
We created a limited dataset for each place based on the photo location information. The tourist attractions, nearby places and coordinates that were too far from the venues were used to exclude irrelevant photos from the limited dataset. The goal was to be able to create more relevant matches between the events and photos.
In the first run, searching for photos that matched the relevant events was made against the datasets in which the photos were limited based on the places.
The run consisted of a set of queries that include matching the artist name and the time of the event, the venue name and the time of the event, and the event name and the time of the event with the metadata (title, tags, description and time taken) of the photos in the dataset.
The goal of this run was to get a set of highly relevant matches between events and photos.
In the second run we used the results of the first run, but we created additional searches for the total dataset of photos for finding more relevant photos.
Event names without time restriction were queried against the metadata of photos. In the case of Parc del Forum, event names were quite unique such as Primavera Sound 2009 and the queries found relevant images. In the case of Paradiso the name of events were often same as the artist that were performing in the event. If time restriction was not used together with the event name, quite a lot of irrelevant photos were attached to the events. We used this query only in the case of Parc del Forum.
The event profiles and their tag clouds were enhanced with the results of the first run, namely the tags from the photos that were found relevant to the event. In this phase, the event profiles consisted of the event name, venue, city, artists, genre, band member information, and the photo tags from the previous run. Apache Solr7 and Lucene8 were used in free-text indexing and searching the textual photo metadata, namely tags and photo descriptions. The photo index was searched with the information in the event profile. The Lucene score limit for accepted result was set relatively high (i.e. 0.5) so that the irrelevant photos would be left out. To further increase the relevance the searches were run on the limited datasets of the Barcelona photos and the Amsterdam photos as described in chapter 2.1.
The results of our submitted runs can be seen in the table 1. The
evaluation measures are described in [
F-score 57,96 68,67 0,5839 0,6782 As expected, the recall of the first run was low due to the use of the limited set of photos, however the photos were quite relevant. Our additions to the second run improved the results and more relevant photos were found.
Our approach of limiting searches to the subset of photos, which was created based on additional information gathered from Linked Data, increased the relevance of photos.
One challenge in the development was the unreliability of the
photo metadata. We could see that the photo timestamps that are
created by different cameras were not always reliable. This made
it difficult to match different images to events using the time
information. The same problem was noted with the GPS
coordinates where even the inherent error in location precision in
city environments is tens of meters [
Solr parameters, like the score parameter, can be adjusted further and more logic can be added to find irrelevant photos especially when the score parameter value is lowered. Other Lucene functionality like MoreLikeThis would also be worth exploring.
The work presented in this paper was partially funded by the OpenSEM project funded by EIT ICT Labs. We would like to thank Onni Ojutkangas, Asko Ollila, Johannes Peltola, Antti Nummiaho and Mika Timonen for code snippets, thoughts and ideas while planning and realizing this task.