This paper describes our contribution to the MediaEval 2014 task on the Synchronization of multi-user Event Media (SEM). We propose two multimodal approaches that employ both visual and time information for the synchronization of different images galleries and for the detections of sub-events. The methods prove robustness in the determination of time o sets with accuracy of up to 87%.
A multifaceted view of a social event can emerge when di erent people capture di erent perspectives of the same event and a compilation of all images is created. While it is typically easy to get an overview of a single image gallery, it is much more di cult to synchronize the content of two or more collections. In general, there is no guarantee that timestamps, location information or textual descriptions associated with images are correct.
In our contributions to the SEM task [
We employ AHC for both time o set calculation and subevent detection. We rst cluster all images of the dataset using the MPEG7 Color Structure Descriptor (MPEG7-CS). At the very lowest hierarchy level clusters of visually highly similar images are generated. We sort these pairs of images in ascending order according to their dissimilarity level. We consider such pairs of images identical if: 1) the images originate from di erent galleries and 2) the dissimilarity distance does not exceed a prede ned threshold. Images, representing di erent galleries, are considered as entry points for the synchronization of the corresponding gallery. We process the sorted pair list until we are able to build a transitive list of entry points for all galleries presented in the full dataset or we reach the end of the list. Eventually, all galleries are time aligned according to the provided reference collection using the corresponding entry points.
A higher hierarchy level of AHC already provides a
reliable base for visual-based detection of sub-events. In order
to avoid the building of broad clusters, we employ a strict
cuto threshold in combination with the Ward method [
For this approach we employ a modi ed version of the
algorithm presented in [
In order to synchronize the dataset, we rst cluster all images using the XMeans algorithm. Following, we consider the average deviation of the reference image timestamps to all other images of a collection that share a common cluster as o set for this image collection. If there are less than two reference images in a cluster, we use the available corrected timestamp of non-reference images which already have an o set from another cluster. For sub-event detection, we employ XMeans clustering using JCD or the corrected capture times as features. 3.
The SEM development dataset contains 304 Flickr images from the London Olympic Games 2012. The images are arranged in 10 galleries and represent 59 sub-events in total. Experiments on the development dataset show signi cant di erences in the precision of detected time o sets between the two approaches. While, the AHC-based approach in combination with MPEG7-CS achieves 18:5 seconds deviation in average over the 10 galleries, the XMeans-based and the JCD feature obtain only 2216:4 seconds in average.
Additionally, we compare the performances of purely timebased clustering (after considering the time o sets), visualbased clustering, and the combination thereof using the AHC approach. We measure the performance by means of harmonic mean (F1-score) of recall and precision and Normalized Mutual Information (NMI) measuring the goodness of clustering of retrieved events. The results achieved show that both the time-based and the visual-based clustering result in over-segmentation of the underlying events (90+ detected sub-events vs. 59 ground truth events) and high NMI scores. The combination of visual and time information outperforms the visual-based approach and signi cantly reduces the number of detected sub-event clusters (see Table 1). Noteworthy is the observation that with both approaches, the time-based detection of sub-events outperforms the corresponding visual-based approach in terms of F1 (at higher over-segmentation costs).
We submitted ve runs for the nal evaluation (see Table 2 for the con gurations). Tables 3 and 4 summarize the corresponding results for the synchronization and for the subevent detection task. Results on the synchronization task are reported in terms of precision (percentage of synchronized galleries with a misalignment lower than 30 minutes), and accuracy (closeness of detected o set to real o set, normalized with respect to the maximum accepted time lapse of 30 minutes). The results achieved con rm our experiments on the development dataset: the AHC-based approach in combination with the MPEG-7-CS clearly outperform our XMeans-based approach. Although both datasets contain approximately the same number of galleries (35 Vancouver, 37 London) they perform di erently. The Vancouver dataset was highly successfully aligned within the maximum accepted time lapse of 30 minutes with a precision of 94%. By contrast, the London dataset achieves a good overall performance by means of an accuracy of 87% at a signi cantly lower precision level of 47%. The results on the sub-event detection task are ambiguous. Overall, the AHC-based approach tends to detect a signi cantly larger number of subevents than the XMeans-based approach. Nevertheless, both approaches result in high Random Index (RI) scores which re ects the purity of the detected clusters. While in general high RI scores may also be the result of strong oversegmentation, the number of detected clusters with our runs di er signi cantly. 4.
In this paper we presented two multimodal approaches for the synchronization of multi-user galleries and for the detection of sub-events. The results obtained on the SEM datasets indicate the potential of the combination of visual and time information for the tasks. An open issue is the detection of sub-events that are visually highly similar and that take place in a short time period.
This work has been partly funded by the Vienna Science and Technology Fund (WWTF) through project ICT12-010, by the Carinthian Economic Promotion Fund (KWF) under grant KWF-20214/22573/33955, and by the iAD center for Research-based Innovation (project number 174867) funded by the Norwegian Research Council.