The event synchronisation task addresses the problem of aligning photo streams from di erent users temporally and identifying coherent events in the streams. In our approach, we rst determine the visual similarity of image pairs. We determine visual similarity based on full matching of SIFT descriptors and based on VLAD, and compare the use of the two sets of similarity scores. We then build a nonhomogeneous linear equation system constraining the time o sets between the galleries based on these matching pairs and determine an approximate solution. Event clusters are initialised from subsequent and visually similar images, and clusters are merged if their temporal proximity and the maximum similarity of their members is high enough.
The event synchronisation task addresses the problem of
aligning photo streams from di erent users temporally and
identifying coherent events in the streams. This paper
describes the work done by the JRS team for the two subtasks
of determining the time o sets of galleries and clustering the
images into events. Details on the task and the data set can
be found in [
In our approach, we rst determine the visual similarity
of image pairs. We determine visual similarity based on full
matching of SIFT [
The computation of the image similarities between the images of each gallery is based on SIFT descriptors. All images of each gallery were rst downscaled from HD to SD. Subsequently, up to 500 SIFT key points and descriptors were extracted from each image.
For similarity calculation based on nearest neighbor matching of SIFT descriptors, each raw SIFT descriptor of the source image is assigned to its nearest neighbour (based on Euclidean distance) descriptor in the target image. These assignments are validated by a homography extracted with the maximum number of descriptors supporting a consistent homography.
For the extraction of the image similarities based on the compact feature representation VLAD the same extracted SIFT key descriptors were used. In order to compute the VLAD signature of each gallery image we used the VLFeat 1 open source library. We reduced a global visual vocabulary with about 300,000 descriptor cluster using k-means clustering to 256 visual words. The descriptors for building the vocabulary have been extracted from a news data set of the TOSCA-MP project2. Based on sum of squared errors the similarities between the VLAD signatures and thus the image similarities within the test sets were calculated.
For a pair of images (Ii; Ij ), VLAD yields distances diVj , which are transformed into similarities siVj =
V diVj ; if diVj < V 0; otherwise; where V is a threshold for the maximum distance. The SIFT similarity siSj is determined as siSj = (
jPijj max(0; min(jPij;jPjj) 0; otherwise;
S); if jPij j p (1) (2) where Pi are the key points in each of the images, Pij is the set of matching key points, p is a threshold for the number of matching key points and S is a similarity threshold. We use all similarities above zero to formulate constraints on the time o sets of the galleries. Optionally, the GPS information of the images (if available) can be used, setting the similarity to zero, if the deviation in longitude or latitude is above a threshold G (in degrees).
For N galleries G1; : : : ; GN , we can assume without loss of generality that G1 is the reference gallery. We aim at obtaining a list of time di erences D = ( 2; : : : ; N ), where i is the time o set between galleries Gi and G1. As the underlying assumption in this task is that the o set between two galleries is constant over time, each pair of matching images adds one constraint of the form p q = ij , where p; q are the galleries containing images Ii; Ij respectively, and ij is the time o set determined from time stamps of the matching images. Note that 1 is by de nition 0. We can then reorganise our constraints into an overdetermined equation system gN (i) gN (j) 3 2 . . .
. .
. g2(k) g2(l) gN (k) gN (l) (3) where gn(i) is a binary function, yielding 1 if I 2 Gn, 0 otherwise. In order to deal with outliers, we iteratively solve the equation system, and remove up to 10% of the largest outliers. In each iteration, we use the Jacobi method to solve the equation system. 2.2
We initialise the event time line by grouping subsequent images, which have visual similarity (sV or sS) above zero. This will oversegment the event time line. In a next step, we start regrouping these events based on visual similarity and (optionally) temporal proximity. The distance between two events i; j is determined as 2 3 2 ij 3 . . 7 6 . 7 = 6 . 7 5 4 . 5 4 . 5
N kl diEj = tmax(1; jti tj j ) tmin (4) where ti is the mean time of images in event Ei, t is a weight for using time information, is the similarity threshold used (S or V) and skl is the visual similarity between a pair of images of which one belongs to Ei and the other to Ej . Two events are merged if diEj < merge, where merge has been set to V + 0:15.
We submitted four runs, with the parameters listed in Table 1. One observation of the experiments of the test set is that full matching of SIFT descriptors is better for determining gallery o sets, which needs to nd the single most similar image from the other gallery. In contrast, the event clustering needs a more global notion of similarity, which is well covered by VLAD. Thus we used VLAD similarities for event clustering in all the runs. The results for synchronisation are shown in Figure 1, and those for clustering in Figure 2.
As already expected from the experiments on the development set, VLAD is not discriminative enough for determining the image pairs for synchronisation, thus the results of run 1 are much worse than the others. Our method only manages to sychronise a fraction of the galleries correctly, however, if a gallery is sychronised, the accuracy is rather high. The results for the London data set are clearly better than those for the Vancouver set. We think that this is 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0 precision Vancouver accuracy Vancouver precision London accuracy London not so much related with the similarity to the development set, but rather with the high visual similarity in Winter Olympics (e.g., all ice based competitions have high similarity). For clustering, the di erences are not so clear, for runs 2 and 3 the Vancouver results are even better than the London ones according to Jaccard index and F-measure. In general, the Rand index shows a quite di erent picture than the other two measures.
The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n 610370, \ICoSOLE { Immersive Coverage of Spatially Outspread Live Events" (http://www.icosole.eu/).