We present a novel solution to the MediaEval 2014 Event Synchronization Task: Synchronization of Multi-User Event Media (SEM). The framework is based on a probabilistic graphical model. Thanks to the simple topology of the graph, the estimation of the true temporal displacement among multiple photo collections can be performed e ciently through exact inference. The underlying tness function is de ned in a exible way, for which it is possible to integrate easily new information (e.g., text tags or social network data). The exibility makes the framework suitable and adaptable to cope with many real situations. The method is evaluated on two datasets obtaining an overall accuracy of more than 85% in both cases.
The problem of photo stream synchronization has been investigated in little work in the current literature. Nevertheless, it represents an open and attractive research topic, especially if one considers its potential applicability to the context of online photo sharing communities.
Indeed, a novel task on this issue has been introduced in MediaEval 2014 [4], where the scenario considered is represented by a number of users attending the same event and taking photos and videos with di erent non-synchronized devices. The goal of the task is twofold.
The synchronization consists of nding the correct temporal o set of each photo collection, denoted as P 1 = (p11; p2; 1 : : : ; p1N ), with respect to a reference gallery, namely P 2 = (p12; p22; : : : ; p2M ), where N and M correspond to the lengths of the two streams.
Once the sequential chronological order of all pictures is restored, the clustering phase is evaluated based on the number of sub-events detected as well as on the quality of the obtained groupings.
The proposed framework for synchronization is based on a probabilistic graphical model. Each temporal displacement can be uniquely identi ed by a set of nearest-neighbor picture pairs across the two photo sets. Fig. 1 shows an example of this concept, where each image in the collection to be synchronized can be compared with the nearest image in the reference gallery (notice that N will be often smaller than M but this is not always true).
Given the set of all possible temporal displacements, namely f Ti : i = 1; : : : ; Qg, and the sequence of all correspondences between pictures given the o set Ti, namely x Ti = (x1 Ti ; : : : ; xj Ti ; : : : ; xNTi ), where xj Ti identi es the picture in the reference P 2 associated with image pj1 given Ti, the synchronization task can be cast into an optimization problem for nding the best o set T . In other words, T = arg max f (x Ti )
Ti
(
Now it is possible to de ne an undirected graphical model through a sequence of observed nodes y = (y1; : : : ; yj ; : : : ; yN ), where yj refers to the image pj1 in P 1, and a sequence of latent variables x = (x1; : : : ; xj ; : : : ; xN ), whose admissible values are de ned over the set X. The edges of the model are of two kinds: links between nodes in x and y, that compare the similarity across photos of the two galleries, and links between nodes in the same sequence x, which take into account the temporal structure of the collections. Fig. 2 summarizes the graphical model described so far.
The joint distribution associated with the graph can therefore be factorized in the following form: p(x) = 1 Y
Z
j
(xj ; xj+1) Y
k
(xk; yk)
(
DHmax(k) DS(xk; yk) DSmax(k)
+ DT (xj ; xj+1) o
DTmax(j)
DG(xk; yk) io
DGmax(k)
(
DT (xj ; xj+1) = jtyj+1
txj+1 j + jtyj
txj j
(
As far as the clustering challenge is concerned, the kmeans algorithm is used to nd the natural grouping of images, see Section 3.
The learning of the parameters described in the previous section is carried out by performing an optimization of the joint distribution over the parameter solution space and using the training dataset made available by [4]. The estimated values for the parameters are = 2:4249; = 0:9509; = 0:9594 and = 3:8597. Once the training phase has completed, one can start to synchronize any pair of galleries.
Results obtained in the SEM task for synchronization are summarized in Table 1. In both datasets, the accuracy of synchronization is greater than 85%, which proves the e ectiveness of the proposed algorithm. Nevertheless, the precision obtained is quite poor since on average only one fourth of the photo collections are correctly synchronized. The low performance is mainly due to forcing associations between images of the two galleries. In many cases, there are pictures that have no correspondence in the reference set.
Once the galleries are synchronized, the corrected temporal information can be exploited to perform clustering. At this purpose, the k-means algorithm is used over three di erent combinations of features. The rst con guration consists of the concatenation of Global Color Structure Descriptors (CSD) [1] with Local Binary Patterns [5]. The second con guration consists only of 6 CSD values obtained by performing PCA reduction on the original CSD descriptor, while the last set up is obtained by adding the temporal information to the second con guration. Table 2 shows the results associated with the three di erent runs. In general, the inclusion of the temporal feature doesn't signi cantly increase the performance. But it's evident that if one is able to carry out a precise synchronization, then temporal information becomes a very reliable feature to perform clustering. This is con rmed by the fact that the results are slightly more than 10% in terms of F-measure and low-level visual features are therefore not su cient.
Future work will be devoted to increasing the synchronization precision in order to allow the exploitation of the temporal component. One possible approach consists of modifying the structure of the graphical model, such that new binary latent variables take into account the possibility of having no associations between photos.