This paper describes the participation of CERTH in the “Social Event Detection Task @ MediaEval 2011”, which aims at discovering social events in a large photo collection. The task comprises two challenges: (i) identification of soccer events in the cities of Barcelona and Rome, and (ii) identification of events taking place in two specific venues. We adopt an approach that combines spatial and temporal filters with tag-based location classification models and an efficient photo clustering method. In our best runs, we achieve F-measure and NMI scores of 77:4% and 0:63 respectively for Challenge 1, and 64% and 0:38 for Challenge 2.
Subsequently, a finer-grained classifier is employed for selecting only the photos that are related to the topic/entity of interest. For Challenge 1, a soccer classifier was created, while a venue classifier was employed for Challenge 2. Both classifiers rely on an approach similar to the one described for the city classification. In both cases, appropriate tag models (soccer model and venue models, one for each of the venues specified by the task) were used that will be further described in Section 3. N M I 0.2516 0.2629 0.2527 0.2646 0.3796
P 51.36 50.58 51.27 50.58 54.31
R 48.85 48.85 48.85 48.85 77.90
We define a single event by a date-place combination. For that reason, we first enumerate all unique dates that appear in the set of photos collected from the photo filtering step described above. For each unique date, we consider a distinct event, except for the dates for which there are photos classified to more than one city of interest. For these dates, one distinct event is considered for each different city, with which at least one photo is associated. At the end of this step, a list of events is available and each event of this list is associated with a set of photos.
Each event produced by the event partitioning step is
enriched by making use of the metadata of the photos
associated with it. A first expansion is carried out by adding
photos of the same user at the same day of the event. Next,
photos with geotagging information that are located in the
vicinity of the event (within a radius of 200m) are also
added to the event under consideration. Finally, an
additional list of photos related to the event are discovered by
means of clustering the photo collection and selecting the
photos of the same cluster under the constraint that their
creators/owners are already associated to the event through
at least one photo. The photo collection is clustered by
means of a community detection scheme that is applied on
a visual, tag or hybrid similarity graph [
We present a set of 10 experiments that evaluate the
performance of the system under a variety of configurations.
Tables 1 and 2 summarize the results obtained from the
official submission of the five runs to Challenges 1 and 2
respectively. In both tables, the run number along with the
selected parameters are listed together with the achieved
performance scores, Normalized Mutual Information (N M I),
Precision (P ), Recall (R) and F-measure (F ), which are
described in [
The first parameter (p1) pertains to the tag model used for filtering out irrelevant photos. Two different soccer tag models were used for Challenge 1, i.e. p1 2 fm1;b; m1;+g, where m1;b is the baseline soccer tag model containing generic soccer tags as well as tags consisting of spanish and italian football club names. The extended tag soccer model (m1;+) additionally contains alternative team names (e.g. “Blaugrana” for Barcelona FC) and stadium names. For Challenge 2, a similar selection was available: the baseline venue tag model (m2;b) consisted of few tags with generic music event terms (e.g. “concert”, “gig”) as well as the names of the two venues of interest. The extended model (m2;+) was enriched with the names of the bands playing in these venues in May 2009 that were retrieved by use of the last.fm API. The second parameter (p2) regards the use of description along with the photo title/tags (p2 2 ftt; ttdg), ttd denoting the use of description in addition to title/tags (tt). The third parameter (p3) regards the use of clustering for event expansion (p3 2 f∅; T; V; Hg), where the options of tag-based (T ), visual (V ), hybrid (H) clusters (produced by graph-based clustering on graphs comprising both tag-based and visual similarities) or no clusters at all (∅) were available. ttd tt tt tt ttd
Finally, the use of image clustering appears to be of limited value to the system. In Challenge 1, there is a modest improvement in the obtained NMI when using the tag-based clusters (run 3 versus run 2), and a marginal improvement when usign the visual clusters. However, a slight drop in precision is observed when using the visual clusters (run 4 versus run 2). Similar results are obtained for Challenge 2. We attribute this result to the fact that most of the potential gains of clustering are already captured by the user-based event expansion (used in all runs).
In conclusion, the experiments indicate the importance of textual metadata of photos in combination with rich domain knowledge for the effective detection of events in large photo collections. In the future, we plan a more comprehensive analysis of false positives and false negatives in order to further improve the system performance.
This work has been supported by the GLOCAL EC project under contract number FP7-248984.