=Paper=
{{Paper
|id=Vol-1263/paper32
|storemode=property
|title=Synchronization of Multi-User Event Media (SEM) at MediaEval 2014: Task Description, Datasets, and Evaluation
|pdfUrl=https://ceur-ws.org/Vol-1263/mediaeval2014_submission_32.pdf
|volume=Vol-1263
|dblpUrl=https://dblp.org/rec/conf/mediaeval/ConciNM14
}}
==Synchronization of Multi-User Event Media (SEM) at MediaEval 2014: Task Description, Datasets, and Evaluation==
https://ceur-ws.org/Vol-1263/mediaeval2014_submission_32.pdf
Synchronization of Multi-User Event Media (SEM) at
MediaEval 2014: Task Description, Datasets, and
Evaluation
Nicola Conci Francesco De Natale Vasileios Mezaris
DISI - University of Trento DISI - University of Trento CERTH - ITI
Trento, Italy Trento, Italy Thermi, Greece
nicola.conci@unitn.it francesco.denatale@unitn.it bmezaris@iti.gr
ABSTRACT tion information is rarely available. In some other cases,
In this paper we provide an overview of the Synchroniza- images might be processed offline for post-production, thus
tion of Multi-User Event Media (SEM) Task that is part losing the correct temporal information. In such cases, cre-
of the 2014 MediaEval Benchmark for Multimedia Evalua- ating a single timeline could turn out to be complicated and
tion. The SEM task is presented this year for the first time challenging, with a concrete risk of representing the event in
in MediaEval, and poses a new challenge, namely the tem- a misleading way.
poral alignment of a series of photo galleries that relate to
the same event but have been collected by different users. 2. TASK DESCRIPTION
Besides aligning the pictures on a common timeline, partici- In our scenario we imagine a number of users (10+) at-
pants are also required to detect the sub-events attended by tending the same event and taking photos and videos with
the users and to group the pictures accordingly. The task different non-synchronized devices (smartphones, handheld
is validated on two different datasets related to the 2010 cameras, DSLRs, tablets). Each user contributes to the task
and 2012 Winter / Summer Olympic games, each dataset with one gallery, which includes an arbitrary number of pho-
comprising a variable number of pictures, galleries, and sub- tos, possibly covering just a part of the event, with variable
events. density of acquisitions (single photos are also possible). As-
suming that these users would like to merge their photo
1. INTRODUCTION galleries in a single event-related collection, the best tem-
poral alignment among the galleries should be found, so as
Content creation is more and more a collective experi-
to correctly report and preserve the temporal evolution of
ence. People attending large social events (a soccer match,
the event. Furthermore, considering the high variability in
a concert), but also personal-scale ones (a wedding, a birth-
terms of acquisition devices, we cannot expect the clocks of
day party) collect dozens of photos and video clips with
each device of the same user to be synchronized, neither in
their smartphones, tablets, cameras, and more recently so-
terms of precision, nor in terms of the time zone set by the
cial cameras. Such information is later exchanged in a num-
users. Furthermore, in some cases, also the location data
ber of different ways, including shared repositories, clouds,
could be unavailable (not all devices have a GPS onboard),
social networks, etc. In this way, different media galleries
further reducing the available information about the cap-
are made available to each other, making it possible for any
tured event. In view of creating a single timeline, these fac-
user who attended, or is simply interested to the event, to
tors may considerably hinder the quality of the alignment,
create his own view of it through summaries, stories, person-
thus different solutions should be envisaged, encompassing
alized albums [1][2]. However, such a large amount of data
the joint analysis of temporal data, position information,
turns out to be unstructured and heterogeneous and, even if
and visual similarity.
it would be possible to collect it on the same hard drive, the
Therefore, the SEM task expects teams to provide the
variability in terms of content, naming, archiving strategies
estimated time offset between different galleries of pictures
makes it impossible to organize all the event-related material
collected by different users and cameras. The goal can be
in a simple yet effective manner.
summarised as follows: given a set of image collections (gal-
In this respect, a major issue is the need of aligning and
leries) taken by different users/devices at the same event,
presenting the various media galleries captured during an
find the best (relative) time alignment among them at gallery
event in a consistent way [3]. As a matter of fact, the time
level, and detect the significant sub-events over the whole
and location information attached to the captured media
event collection.
(timestamp, GPS) can be wrong, inaccurate or even miss-
ing (for instance, due to wrong setting of the clock/calendar,
different time-zone, modification or removal of tags). Sim- 3. DATASETS
ilarly, this is also a common situation in historical events For this challenge we make available two different datasets,
and photo archives, where timestamps and especially loca- consisting of a collection of images gathered from Flickr
and made available under Creative Commons license. Both
datasets refer to well known and structured sport events,
Copyright is held by the author/owner(s).
MediaEval 2014 Workshop, October 16-17, 2014, Barcelona, Spain namely the Olympic Games held in London in 2012 and the
. Vancouver Winter Olympic Games of 2010. We have cho-
Figure 1: Sample images taken from the two datasets.
sen to work with these two events because on the one hand TP
JI = (4)
they exhibit a clear and organized schedule with precise tim- TP + FP + FN
ing. On the other hand they still exhibit a high variabil-
ity in terms of visual content, due to the common features 2P R
F1 = (5)
across different competitions in the same discipline, as well P +R
as strong similarities in the environments, in which the pic- Precision measures the number of galleries (M ) over the
tures are collected, making the synchronization a non-trivial total number of galleries (N − 1, excluding the reference),
task. As far as this task is concerned, the images within a that have been correctly synchronized, namely those gal-
gallery are consistent in terms of timestamp, and might in- leries, for which the alignment error with respect to the ref-
clude the GPS information. Therefore the temporal offsets erence gallery, is below a threshold. With the accuracy we
are at gallery level thus assuming that every user uses one instead evaluate the capabilities of the teams in minimizing
single device for acquisition. the average time lapse calculated over the M synchronized
The dataset collected from the London Olympics includes galleries, normalized with respect to the maximum accepted
2124 images, divided into 37 galleries. The first gallery com- time lapse.
prises a subset of the data provided in the development set The synchronization task provides a basis for the cluster-
and is defined as the reference gallery. The dataset col- ing task. Once the galleries are synchronized, it is possible
lected from the Vancouver Winter Olympic Games includes to cluster the whole event collection to detect sub-events
1351 pictures representing most of the competitions, divided occurring within the entire event, for instance, the single
into 35 galleries with a variable number of pictures in each competitions, or the ceremonies of the different disciplines.
gallery. Also in this case, the first gallery is set as the refer- Sub-events are defined in a neutral and unbiased way (e.g.,
ence. Fig. 1 shows a few samples of the two datasets. making reference to the calendar/schedule of the event) and
coded into the ground truth. We measure the performance
4. METRICS AND EVALUATION of the sub-event clustering over the whole synchronized col-
Two objective metrics will be used to evaluate the results: lection of media. In this case, we use the three performance
indicators reported above, namely RI, JI, and F1. In the
• time synchronization error formulation we define a true positives (TP), in case two im-
• sub-event detection error ages related to the same sub-event are associated the same
cluster, and the true negative (TN), when two images asso-
As far as the first metric is concerned, the goal of the par- ciated to different sub-events are assigned to two different
ticipants is to maximize the number of galleries for which the clusters). False positives (FP) occur instead when two im-
synchronization error is below a predefined threshold, and ages are assigned to the same cluster although belonging to
to minimize the time shift of those galleries. The synchro- different sub-events.
nization error for a gallery Gi with respect to the reference
∗ ∗
Gr is defined as ∆Eir = ∆Tir − ∆Tir , where ∆Tir is the
delay between Gi and Gr calculated on the ground truth.
5. ACKNOWLEDGMENTS
The threshold ∆Emax depends on the duration of the sub- This work was supported in part by the EC under con-
events in the dataset, and represents the maximum accepted tract FP7-600826 ForgetIT. We would like to thank Anas-
time lapse within which we consider a gallery as reasonably tasia Ioannidou, Alain Malacarne, and Alessio Xompero for
well-synchronized. their precious help in collecting the images for the dataset.
As far the metrics for evaluation are concerned, we have
considered for the temporal alignment the precision (Eq. 1) 6. REFERENCES
and accuracy (Eq. 2). For the quality of the clustering, we [1] M. Broilo, G. Boato, and F. De Natale. Content-based
use the Rand Index (RI), as from Eq. 3, the Jaccard index synchronization for multiple photos galleries. In
(JI) Eq. 4 , and the F1 score Eq. 5, where P and R represent Proceedings - International Conference on Image
the Precision and Recall, respectively. Processing, ICIP, pages 1945–1948, 2012.
[2] G. Kim and E. P. Xing. Jointly aligning and
M Card (∆Eir < ∆Emax ) segmenting multiple web photo streams for the
P recision = = (1) inference of collective photo storylines. In Proceedings
N −1 N −1
of the 2013 IEEE Conference on Computer Vision and
PN −1 Pattern Recognition, CVPR ’13, pages 620–627, 2013.
i=1 ∆Eir
Accuracy = 1 − (2) [3] J. Yang, J. Luo, J. Yu, and T. Huang. Photo stream
(N − 1)∆Emax
alignment and summarization for collaborative photo
TP + TN collection and sharing. Multimedia, IEEE Transactions
RI = (3) on, 14(6):1642–1651, Dec 2012.
TP + TN + FP + FN